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Abstract 


Project  managers  typically  set  three  success  criteria  for  their  projects:  meet 
specifications,  be  on  time,  and  be  on  budget.  However,  software  projects  frequently  fail  to  meet 
these  criteria. 

Software  engineers,  acquisition  officers,  and  project  managers  have  all  studied  this  issue 
and  made  recommendations  for  achieving  success.  But  most  of  this  research  in  peer  reviewed 
journals  has  focused  on  the  private  sector.  Researchers  have  also  identified  software  acquisitions 
as  one  of  the  major  differences  between  the  private  sector  and  public  sector  MIS.  This  indicates 
that  the  elements  for  a  successful  software  project  in  the  public  sector  may  be  different  from  the 
private  sector. 

Private  sector  project  success  depends  on  many  elements.  Three  of  them  are  user 
interaction  with  the  project's  development,  critical  success  factors,  and  how  the  project  manager 
prioritizes  the  traditional  success  criteria.  High  user  interaction  causes  high  customer 
satisfaction,  even  when  the  traditional  success  criteria  are  not  completely  met.  Critical  success 
factors  are  those  factors  a  project  manager  must  properly  handle  to  avoid  failure.  And  priorities 
influence  which  success  criteria  the  project  manager  will  most  likely  succeed  in  meeting. 

Through  a  survey  of  software  project  managers  at  two  IJSAF  software  development 
organizations,  my  research  discovered  the  following: 

1)  Air  Force  software  project  managers’  top  priority  is  fulfilling  requirements, 

2)  User  interaction  during  the  software  life  cycle  strongly  influences  user  satisfaction 
with  the  final  product,  and 

3)  Air  Force  and  private  sector  projects  share  many  of  the  same  critical  success  factors 
for  nonweapon  systems,  but  there  are  still  some  sharp  differences. 
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THE  GENERALIZ ABILITY  OF  PRIVATE  SECTOR  RESEARCH 


ON  SOFTWARE  PROJECT  MANAGEMENT  IN  TWO  USAF  ORGANIZATIONS: 

AN  EXPLORATORY  STUDY 


I.  Introduction 


Background 

Professional  project  management  deals  with  multi-million  dollar  endeavors  that  can  take 
years  to  finish.  Space  programs,  buildings,  bridges,  dams,  pharmaceutical  products,  jets,  and 
weapon  systems  are  all  traditional  examples  of  programs  that  require  project  management. 
During  the  last  three  decades,  software  projects  have  joined  this  group  and  challenged  project 
managers  with  their  intangible  existence  and  complexity. 

The  proper  management  of  software  projects  makes  the  difference  in  achieving  the  three 
traditional  success  criteria:  completing  the  project  on  budget,  implementing  the  project  on 
schedule,  and  fulfilling  all  of  the  users’  specifications.  Nonetheless,  project  success  is  a  difficult 
outcome  to  nail  down.  The  user  of  this  software  product  must  be  satisfied  with  the  software 
product  and  use  it  before  recognizing  it  as  a  success,  regardless  of  how  well  the  project  manager 
achieves  other  criteria.  The  user’s  attitude,  involvement,  and  participation  with  the  project’s 
development  greatly  influence  satisfaction.  Likewise,  additional  success  factors  are  critical  in 
reaching  the  traditional  success  criteria  (such  as  support  from  senior  management,  effective 
management  of  risk,  and  effective  leadership). 

Due  to  difficulties  in  attaining  these  criteria,  literature  has  noted  many  projects  that  have 
failed  some  or  all  of  these  criteria.  For  example,  the  Standish  Group  Report  (1995)  discussed 
their  research  for  365  companies  on  8,380  software  projects.  31%  of  the  projects  were  cancelled. 
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53%  of  the  completed  projects  were  189%  over  budget.  And  only  16%  of  the  completed  projects 
actually  finished  on  time  and  on  budget. 


Meanwhile,  the  Department  of 

Table  1 

Defense  is  acquiring  and  maintaining 


weapon  systems  that  depend  on  these 
software  projects  to  be  successful  (Table  1). 
For  example,  the  defunct  Crusader  Artillery 
used  1,800,000  lines  of  code,  and  the  nearly 
complete  F-22  uses  1,960,000  lines  of  code. 
The  success  of  these  software  systems 
impact  the  survival  of  Americans  and 
mission  accomplishment.  Likewise, 


Weapon  System  Software  Sizes 


Weapon  System 

Source  Lines  of 
Code 

Ml  Tank 

600,000 

Scout  /  Cav 

1,000,000 

M2  Infantry  Fighting  Vehicle 

1,560,000 

Crusader  Artillery 

1,800,000 

F-22 

1,960,000 

Aegis 

2,840,000 

Note.  From  “Curing  the  Software  Requirements  and  Cost 
successful  nonweapon  software  systems  Estimating  Blues,”  by  M.  Nelson,  J.  Clark,  and  M.  A. 

Spurlock,  1999,  Program  Manager,  17,  p.  54. 

increase  the  military’s  productivity  and  communications  capabilities.  For  example,  the  USAF 
plans  to  connect  700  software  systems  with  the  Air  Force  Portal  so  members  have  access  to  all 
the  information  they  require  for  their  duties. 


Problem  Statement 

In  response  to  these  stories  of  software  crises  and  project  failures,  many  professional 
journals  (such  as  Cross  Talk  for  DoD  software  engineers)  and  peer-reviewed  literature  have 
published  articles  on  software  projects  and  project  management  in  general.  However,  there  is 
little  published  research  that  focuses  specifically  on  software  projects  within  the  military  and 
federal  government;  practically  all  literature  deals  with  software  projects  in  the  private  sector. 
Military  project  management  for  software  development  and  maintenance  is  a  highly  relevant 
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issue  because  the  military  has  had  an  equally  difficult  time  with  it  as  the  private  sector.  For 
example,  the  Air  Force  spent  $60  million  and  5  years  on  developing  a  new  personnel  information 
software  system.  In  May  of  2001,  the  Air  Force  launched  this  new  system  and  immediately 
started  having  problems,  such  as  preventing  new  military  members  from  receiving  pay.  A  year 
later,  the  software  system  still  suffered  glitches.  At  the  Air  Force  Personnel  Center’s  website 
http://www.afpc.randolph.af.mil/modem/.  technicians  reported  correcting  4,944  problems  with 
the  personnel  system  as  of  August  19th,  2002.  But  on  the  same  day,  they  also  reported  461  more 
complaints  and  bugs  to  pursue.  By  2004,  the  U.S.  Army  plans  to  begin  designing  an  even  larger 
software  system  modeled  off  of  the  Air  Force  system  at  an  expense  of  $500  million. 


Our  military  aircraft  are  also  Table  2 


.  Weapon  System  Software  Dependencies 

increasingly  becoming  dependent  on  _ 


software.  As  Table  2  shows,  in  1960, 

Weapon 

System 

Year 

Percent  of  Functions 
Performed  in  Software 

only  8%  of  the  F-4's  functions  depended 

F-4 

1960 

8 

on  software.  But  within  10  years,  20% 

A-7 

1964 

10 

F-l  11 

1970 

20 

of  the  F-l  1  l's  functions  depended  on 

F-15 

1975 

35 

software,  and,  in  1990,  the  B-2's 

F-16 

1982 

45 

functions  were  software  dependent  by 

B-2 

1990 

65 

65%. 

F-22 

2000 

80 

Note.  From  “Curing  the  Software  Requirements  and  Cost 
These  weapon  systems  and  Estimating  Blues,”  by  M.  Nelson,  J.  Clark,  and  M.  A.  Spurlock, 

1999,  Program  Manager,  17,  p.  55. 


software  systems  are  just  several  of  many  software  products  that  the  Air  Force  depends  on  for 
mission  accomplishment.  Consequently,  research  on  the  generalizability  of  private  sector 
research  to  the  public  sector  can  help  confirm  Air  Force  current  practices  or  identify  the 
necessity  for  different  methodologies  for  software  project  management  in  the  military. 
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Research  Question  and  Investigative  Questions 

Because  the  Air  Force  is  subject  to  the  same  software  project  management  problems  as 
the  private  sector,  the  Air  Force  would  benefit  from  research  generalizable  to  the  military  for  its 
current  and  future  software  project  management  endeavors.  However,  after  conducting  an 
exhaustive  review  of  the  literature,  I  was  unable  to  find  any  substantive  research  on  this  subject 
regarding  military  software  project  management.  Therefore,  I  conducted  an  exploratory  study  to 
learn  about  military  software  project  management  and  break  new  ground  in  this  field.  My 
overall  research  question  asked,  "How  do  the  success  factors  in  software  project  management  for 
the  Air  Force  differ  (if  at  all)  from  the  private  sector?" 

This  question  covered  a  very  diverse  topic,  considering  all  the  possible  factors  that  can  go 
into  successful  project  management.  Therefore,  the  only  factors  studied  were  those  under  the 
project  manager's  control  (i.e.:  no  environmental  factors,  which  are  outside  of  the  project 
manager's  control).  Three  investigative  questions  narrowed  the  scope  of  this  thesis: 

1 .  How  do  Air  Force  software  project  managers  prioritize  the  three  traditional  measures  of 
project  success? 

2.  Does  high  user  interaction  in  Air  Force  software  projects  correlate  with  project  success? 

3.  Are  the  critical  success  factors  for  Air  Force  software  projects  different  from  private 
sector  projects? 

These  investigative  questions  required  a  variety  of  data  to  answer  them.  Software  project 
managers  from  two  Air  Force  units,  "Organization  Alpha”  and  “Organization  Bravo,"  responded 
to  a  questionnaire  to  provide  data  for  the  investigative  questions.  Both  Organizations  Alpha  and 
Bravo  develop  and  sustain  software  for  the  Air  Force,  either  by  developing  in-house,  adapting 
commercial  off  the  shelf  products,  or  supervising  contracted  out  work.  For  the  first  question, 
project  managers  prioritized  success  criteria  under  different  situations,  which  showed  how 
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project  managers  most  frequently  prioritize  their  goals.  For  the  second  question,  project 
managers  used  a  validated  questionnaire  to  score  user  involvement  in  projects  and  their  project's 
level  of  success.  And  for  the  third  question,  project  managers  used  a  survey  to  list  all  the  factors 
that  are  critical  to  project  success.  This  data  was  then  analyzed  through  comparison  with  private 
sector  research  and  statistical  methods. 

Terminology 

This  research  focused  on  software,  as  defined  by  the  Software  Engineering  Process  List 
of  Definitions  (Anon,  2002)  and  derived  from  Peach  (1992):  “Intellectual  creation  comprising 
the  programs,  procedures,  rules  and  any  associated  documentation  pertaining  to  the  operation  of 
a  data  processing  system.  Computer  programs  and  computer  databases.”  All  information 
systems  referred  to  in  the  literature  review  are  under  the  context  of  being  software  systems,  such 
as  computer  databases,  rather  than  non-software  systems,  such  as  a  library  card  catalog.  Articles 
referring  to  project  management  in  general  are  noted  as  general  projects,  rather  than  as 
specifically  software  projects.  This  research  does  not  include  information  technology,  which 
involves  both  software  and  hardware. 

The  distinction  between  "users"  and  "customers"  is  also  important  to  this  thesis. 
According  to  the  Software  Engineering  Process  List  of  Definitions  (Anon,  2002),  a  "user"  (a.k.a. 
end-user)  is  "the  individual  or  group  who  will  use  the  system  for  its  intended  operational  use 
when  it  is  deployed  in  its  environment."  A  "customer"  is  "the  individual  or  organization  that  is 
responsible  for  accepting  the  product  and  authorizing  payment  to  the  developing  organization." 

Both  surveyed  Air  Force  organizations  maintained  identical  software  terminology  and 
agreed  upon  definitions  on  their  websites.  Furthermore,  this  research  excluded  software  on 
weapon  systems.  Military  software  projects  resembling  private  sector  software  projects  were 
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used  for  the  most  accurate  comparison.  Both  surveyed  Air  Force  organizations  developed  and 
maintained  such  software,  and  examples  in  the  questionnaire  also  resembled  software 
applications  and  information  systems  that  a  private  sector  company  would  develop. 

Thesis  Overview 

This  thesis  is  divided  into  five  chapters.  Chapter  I  discusses  the  background  and  the 
problem  of  software  project  success  and  how  this  issue  relates  to  the  military.  Investigative 
questions  narrow  down  the  scope  to  project  success  influenced  by  goals,  user  interaction,  and 
critical  success  factors.  Chapter  II  reviews  peer-reviewed  literature  on  these  constructs. 
Chapter  III  describes  the  development  of  the  questionnaire  and  the  statistical  methods  used  to 
analyze  the  data.  Chapter  IV  presents  the  data  and  its  analysis  for  each  investigative  question 
and  any  conspicuous  differences  between  cross  sections  of  the  data.  And  Chapter  V  interprets 
the  data  analysis  in  answer  to  the  investigative  questions  and  recommends  future  research. 
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II.  Literature  Review 


Overview 

Researchers  and  subject  matter  experts  have  identified  differences  in  methodology 
between  public  and  private  software  acquisitions.  However,  this  study  was  unable  to  find  any 
research  specifically  on  differences  between  private  and  public  sector  project  management. 
Literature  indicates  how  various  factors  influence  the  perceived  success  of  projects  (Table  1). 
Table  3 

Definitions  of  Constructs 


Construct 

Definition 

Communication 

The  provision  of  an  appropriate  network  and  necessary  data 
to  all  key  actors  in  the  project  implementation,  including  the 
user  and  customer  (Slevin  &  Pinto,  1986). 

Critical  Success  Factors 

The  few  factors  that  will  ensure  success  in  a  particular 
business  area  if  the  manager  gives  them  the  necessary 
attention.  Likewise,  if  these  factors  are  disregarded,  the 
endeavor  is  bound  to  fail  (Rockart,  1979). 

Customer  Satisfaction 

The  product  satisfies  all  needs  and  expectations  and  the  user 
is  pleased  with  it,  despite  whether  it  is  on  time,  on  budget,  and 
fulfills  all  specifications  (Pinto  &  Slevin,  1988). 

Project  Success 

Consists  of  both  user  satisfaction  and  the  traditional  success 
criteria  (Pinto  &  Slevin,  1988). 

Traditional  Success 
Criteria 

The  project  must  deliver  a  product  on  schedule,  on  budget, 
and  according  to  performance  specifications  (Pinto  &  Slevin, 

1988). 

User  Consultation 

Communication,  consultation,  and  active  listening  to  all 
impacted  parties  (Slevin  &  Pinto,  1986). 

User  Acceptance 

The  act  of  "selling"  the  final  project  to  its  ultimate  intended 
users  (Slevin  &  Pinto,  1986). 

User  Interaction 

Having  a  user  (or  a  liaison)  participate  in  project  activities,  feel 
involved  with  the  progress  of  the  project  and  its  upcoming 
usefulness,  and  develop  a  confident  attitude  that  the  project  is 
being  managed  well  despite  any  adverse  conditions  (Barki  & 

Hartwick,  1989). 
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Differences  Between  Private  Sector  and  Public  Sector  MIS 


Previous  research  suggests  substantive  differences  between  public  and  private  sector 
management  information  systems  (Bozeman  &  Bretschneider,  1986;  Bretschneider,  1990; 
Caudle,  Gore,  &  Newcomer,  1991).  Management  information  systems  (MIS)  "concerns  both  the 
management  of  information  technology  and  the  use  of  information  technology  for  managerial 
and  organizational  purposes"  (Ives,  1995).  In  regard  to  software  project  management  and  related 
acquisitions,  Bozeman  and  Bretschneider  (1986)  note  public  sector  MIS  “requires  a  protracted 
period  of  testing  and  prototype  development.”  Strong  accountability  is  necessary  because 
government  software  projects  are  open  to  public  scrutiny.  And  Bretschneider  (1990)  adds  that 
private  sector  organizations  evaluate  software  acquisitions  by  the  economic  efficiency  of  their 
performance.  However,  public  sector  software  acquisitions  are  strongly  influenced  by 
procedural  equity  such  as  acquisition  regulations,  government  contracting  rules,  and  Department 
of  Defense  (DoD)  standards.  In  a  review  of  DoD  software  acquisitions,  Jones  (2002a),  a  subject 
matter  expert,  anecdotally  notes  even  more  differences  in  the  public  sector.  In  addition  to  a 
highly  regulated  contractual  procurement  process,  he  also  observes  that  military  procurement 
frequently  has  litigation  challenging  the  successful  bidder  (with  side-effects  like  delays  in  the 
project  schedule)  and  extensive  oversight  and  control  requirements  (resulting  in  documentation 
three  times  larger  than  equivalent  civilian  projects). 

On  the  other  hand,  Devlin  and  Royce  (1994)  comment  in  an  Air  Force  software 
acquisition  book  that  “many  commercial  practices  are  inappropriate  to  most  DoD  software  (the 
glaring  exception  is  DoD’s  MIS  systems  which  only  differ  by  perhaps  their  scale).” 

These  differences  in  MIS  and  software  acquisition  procedures  and  rules  begs  the  question 
whether  public  sector  project  management  of  software  is  distinctly  different  from  the  private 
sector.  If  it  is,  then  military  software  project  management  may  require  a  different  form  of 
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methodology  from  what  the  private  sector  uses.  This  research  sought  to  study  common 
constructs  in  project  management  within  two  Air  Force  software  developing  organizations  that 
might  question  or  affirm  the  generalizability  of  private  sector  research  on  those  two  Air  Force 
organizations. 

Project  Management  Constructs  Under  Review 

While  many  factors  contribute  to  a  project's  successful  completion,  only  some  of  these 
factors  are  under  the  project  manager's  control.  This  research  studied  those  non-environmental 
factors  that  are  most  critical  to  project  success.  These  factors  influence  the  achievement  of 
customer  satisfaction  and  completing  software  projects  on  time,  on  budget,  and  within 
specifications.  The  literature  review  covers  the  following  constructs  toward  achieving  project 
success:  critical  success  factors  (CSFs),  user  interaction  and  customer  satisfaction,  and  success 
criteria  and  priorities. 

Critical  Success  Factors  Research 

J.  F.  Rockart  (1979)  defines  critical  success  factors  (CSFs)  as  the  few  factors  that  will 
ensure  success  in  a  particular  business  area  if  the  manager  gives  them  the  necessary  attention. 
Likewise,  if  these  factors  are  disregarded,  the  endeavor  is  bound  to  fail.  Boynton  and  Zmud 
(1984)  researched  the  CSF  construct  through  case  studies  and  concluded  that  it  is  a  valid 
construct  worthy  of  further  research.  Their  research  indicates  that  two  CSFs  are  managerial 
support  and  “a  positive  relation  and  a  meaningful  dialogue  with  users.”  Various  researchers 
have  since  studied  CSFs  for  project  management  (Delano  1998;  Dobbins  1998;  Pinto  &  Slevin 
1987;  White  &  Fortune  2002).  Each  researcher  has  concluded  with  slightly  different  factors,  but 
all  of  them  have  found  the  user  of  the  project  to  be  a  factor  of  success. 
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Dobbins  (1998)  notes  the  scarcity  of  articles  on  military  project  management  and  then 
details  critical  success  factors  for  defense  acquisition  programs.  He  surveyed  two  groups  of 
defense  program  managers:  those  who  worked  on  software  for  weapon  systems  and  those  who 
worked  on  software  for  information  systems.  Delano  (1998)  made  a  similar  survey  for  the  CSFs 
of  defense  acquisitions  (but  not  specifically  software).  Both  studies  indicate  a  strong  user 
relationship  was  one  of  the  top  six  factors.  Jones  (2002b),  a  subject  matter  expert,  observed 
twelve  CSFs  while  judging  16  of  the  best  DoD  software  projects  for  2001.  Beyond  these  two, 
the  majority  of  project  management  articles  in  peer-reviewed  journals  deal  with  the  private 
sector. 

Pinto  and  Slevin  (1987)  initially  constructed  their  list  of  10  CSFs  from  a  card-based 
survey  of  MBA  students  on  important  factors  for  successful  projects  of  all  types.  Since  then, 
they  have  repeatedly  tested  and  verified  their  CSFs  in  subsequent  studies  (Pinto  &  Mantel,  1990; 
Pinto  &  Slevin,  1988).  Their  studies  indicate  three  factors  that  particularly  influence  obtaining 
customer  satisfaction:  (a)  client  consultation  (actively  listening  to  the  client  and  discussing 
realistic  expectations),  (b)  client  acceptance  (convincing  the  client  the  project  is  worth  the 
expense  and  trouble  of  difficulties),  and  (c)  communication  (updating  the  client,  providing 
feedback,  asking  for  input).  Slevin  and  Pinto  (1986)  designed  a  survey  to  measure  how  well  a 
project  manager  handled  the  10  CSFs  on  a  current  project.  Subsequently,  Pinto  and  Slevin 
(1988)  tested  an  addition  to  the  questionnaire  to  evaluate  the  success  of  a  project  from  the  project 
manager's  perceptions.  Success  was  partly  based  off  of  the  elicitation  of  a  positive  user  attitude 
toward  the  final  product. 

White  and  Fortune’s  study  (2002)  is  one  of  the  most  recent  studies  on  CSFs.  They 
surveyed  project  managers  from  88  different  industries,  but  only  5  worked  in  defense.  Of  the 
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236  respondents,  60  managed  an  "information  technology"  project,  26  had  a  "software 
development"  project,  and  7  had  dealt  with  a  software  project  on  "Year  2000  compliance." 

White  and  Fortune's  questionnaire  offered  a  list  of  19  CSFs  taken  from  literature  (including  Pinto 
and  Slevin,  1987).  They  expanded  the  list  to  23  when  some  of  the  respondents  offered  additional 
CSFs  that  literature  had  not  yielded.  Their  results  indicate  user  commitment  to  the  project  is 
among  the  top  five  CSFs. 

Importance  of  User  Interaction  to  User  Satisfaction 

Various  literature  discusses  the  importance  of  having  user  interaction  throughout  the 
project  life  cycle  to  ensure  user  satisfaction  and  acceptance  of  the  product,  even  when  other 
success  criteria  (such  as  the  schedule)  are  broken. 

Deutsch  (1991)  expanded  the  traditional  success  criteria  (requirements,  schedule,  and 
budget)  for  software  products  to  include  user  satisfaction,  which  is  how  much  users  are  satisfied 
with  the  system's  performance.  This  is  distinct  from  meeting  specifications,  which  may  not 
always  fulfill  the  true  expectations  of  the  user. 

Wateridge  (1999)  also  notes  the  users  and  project  manager  must  meet  periodically  as 
ideas  for  the  project  solidify  and  become  detailed.  User  participation  is  necessary  for  customer 
satisfaction. 

Barki  and  Hartwick  (1989)  broke  user  interaction  down  into  three  phrases:  of  user 
attitude,  user  involvement,  and  user  participation.  Up  to  that  point,  researchers  used  the  words 
interchangeably.  But  Barki  and  Hartwick  noted  the  fields  of  psychology,  marketing,  and 
organizational  behavior  referred  to  "involvement"  as  an  intrinsically  important  and  personal 
event.  Whereas  MIS  researchers  used  "involvement"  just  like  "participation,"  in  which  a  person 
is  doing  a  set  of  activities  with  other  people.  "User  attitude"  describes  how  positively  a  person 
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evaluates  a  new  software  information  system  or  how  that  person  feels  about  it.  Barki  and 
Hartwick  argue  these  three  constructs  should  be  made  consistent  among  other  fields  of  study  and 
be  recognized  as  distinct  constructs. 

To  test  these  constructs,  Barki  and  Hartwick  (1994)  designed  a  questionnaire  to  measure 
user  participation,  attitude,  and  involvement.  They  gathered  59  questions  from  previous 
research,  conducted  a  survey,  and  then  rigorously  tested  it  for  reliability  and  validity  to  make 
sure  the  questionnaire  consistently  evaluated  the  content  they  had  in  mind. 

Barki  and  Hartwick  (1994)  note  there  is  a  moderate  correlation  between  user  participation 
and  user  attitude  and  user  involvement.  When  a  user  participates  in  a  software  project,  the  user 
takes  a  personal  interest  in  the  project's  success,  feels  ownership  for  it,  and  judges  the  project  is 
more  likely  to  succeed. 

Hunton  and  Beeler  (1997)  took  Barki  and  Hartwick's  research  (1994)  another  step  by 
conducting  an  experiment  to  test  this  correlation.  Their  results  indicate  users  are  happier  with  a 
software  system  when  they  are  involved  with  its  development.  When  the  user  feels  he  has 
influence  over  the  input,  then  he  likes  the  output.  Hunton  and  Beeler's  results  suggest  that 
software  project  managers  should  encourage  user  interaction,  especially  when  the  user  can  make 
an  impact. 

Success  Criteria  and  Priorities 

Barki,  Rivard,  and  Talbot  (2001)  conducted  an  experiment  to  test  how  the  user  influences 
the  project's  success.  They  define  success  by  measuring  process  performance  (on  time,  on 
budget)  and  product  performance  (meets  specifications  and  quality  expectations).  They  found 
that  project  managers  were  more  successful  depending  on  how  they  prioritized  to  reach  their 
goals.  Project  managers  were  more  successful  in  reaching  the  goal  of  meeting  process 
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performance  by  focusing  on  the  schedule  and  budget.  But  when  the  project  manager's  primary 
goal  was  system  quality,  then  he  was  more  successful  by  eliciting  user  participation. 

Atkinson  (1999)  argues  that  researchers  should  review  the  traditional  success  criteria  of 
projects,  namely,  (a)  being  on  time,  (b)  on  budget,  and  (c)  meeting  product  specifications.  He 
said  these  three  make  up  the  Iron  Triangle  and  that  projects  seem  to  frequently  fail  because  these 
criteria  for  success  are  inadequate;  they  do  not  properly  describe  standards  forjudging  a 
successful  project.  Success  criteria  should  adapt  to  the  priorities  of  each  project.  For  example, 
life  critical  systems  (such  as  military  fighter  jet  software)  should  have  quality  as  the  overriding 
criteria.  Time  and  cost  are  secondary  issues  in  this  case. 

Lim  and  Mohamed’s  inductive  study  (1999)  notes  that  while  project  managers  measure 
success  off  of  the  traditional  success  criteria,  customers  measure  it  according  to  their  perceptions 
and  satisfaction  with  the  project.  Lim  and  Mohamed  theorize  that  true  project  success  depends 
on  fulfilling  all  of  these  criteria,  with  an  emphasis  on  the  customer’s  perceptions.  A  project 
manager  considers  a  project  a  failure  when  it  goes  over  time  and  over  budget  while  properly 
meeting  specifications.  But  the  customer  is  happy  over  the  long  term  because  the  time  and 
budget  issues  are  only  temporary  stumbling  blocks  in  getting  what  he  wanted.  For  example,  the 
project  managers  for  the  new  F-22  judge  their  success  based  off  of  the  Iron  Triangle.  But  10 
years  from  now,  an  F-22  pilot  will  not  care  whether  the  project  went  over  time  and  over  budget. 
All  the  pilot  cares  about  is  whether  the  plane  fulfills  all  his  expectations  and  makes  him  a 
satisfied  customer.  Wateridge’s  research  (1995,  1998)  indicates  similar  conclusions;  when  the 
project  manager  secures  customer  satisfaction,  an  over  budget  and  overtime  project  is  still  a 
success. 
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Dobbins  (1998)  notes  that  even  though  customer  satisfaction  engenders  project  success, 
program  managers  most  often  measure  projects  by  cost  and  schedule  because  they  must  brief 
oversight  agencies.  This  puts  a  project  manager  into  conflict  when  customer  satisfaction  and 
meeting  specifications  requires  going  over  budget  and  over  time.  Fulfilling  all  these  goals 
becomes  a  difficult  issue  because  the  project  manager  must  prioritize. 

Milosevic,  Inman,  and  Ozbay's  research  (2001)  indicates  that  organizations  using 
strategic  project  management  have  successfully  focused  on  one  of  the  three  traditional  success 
criteria  and  derived  appropriate  results  (i.e.,  schedule-driven  groups  were  always  on  schedule, 
but  quality  and  budget  could  suffer). 

Abdel-Hamid,  Sengupta,  and  Swett  (1999)  studied  this  issue  at  the  project  manager  level 
for  software  projects.  When  a  project  manager's  goal  is  to  finish  a  software  project  on  time  and 
on  budget,  his  behavior  follows  through.  Likewise,  when  a  project  manager's  goal  is  to  be  on 
time  and  provide  a  quality  product,  his  behavior  follows  through  for  timing  and  higher  quality. 

Fowler’s  research  (1999)  notes  that  project  managers  cause  their  projects  to  go  over 
budget  by  forcing  the  project  to  either  meet  an  unrealistic  schedule  or  crashing  the  time  to 
eliminate  slack  and  finish  early.  Ironically,  though,  his  research  also  indicates  that  the  quality  of 
the  product  lowers  because  the  project  manager  cut  comers  and  the  customer  perceives  the 
project  to  be  low  quality  work.  Fowler  concludes  that  project  success  depends  on  customer 
satisfaction,  not  solely  meeting  two  of  the  three  standard  success  criteria.  Khang  and  Myint’s 
study  (1999)  made  similar  conclusions;  crashing  time  causes  the  project  to  go  over  budget  and 
may  still  not  satisfy  the  customer. 
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Delano  (1998)  surveyed  program  managers  on  their  success  criteria  and  discovered  that 
the  highest  ranking  success  indicator  was  meeting  technical  performance  objectives,  and  the 
second  to  last  was  meeting  cost  objectives. 

Rush  (1997)  notes  that,  prior  to  1995,  requirements  and  sometimes  schedules  determined 
costs  in  defense  acquisitions.  These  acquisitions  were  made  in  an  environment  with  a  more 
plentiful  defense  budget.  But  in  1995,  decision  makers  formulated  Cost  as  an  Independent 
Variable  (CAIV)  in  reaction  to  a  decreasing  defense  budget.  CAIV  requires  personnel  to  control 
requirements  and  schedules  with  as  cost  as  the  top  priority.  CAIV  was  first  implemented  in 
1996,  and  it  continues  to  be  emphasized  in  the  latest  guidance  (DoD,  2002). 

Therefore,  while  private  sector  research  indicates  project  managers  prioritize 
requirements  as  their  top  priority,  defense  regulations  have  required  cost  as  the  top  priority  for 
the  last  eight  years  (DoD,  2002).  Nonetheless,  defense  software  projects  are  still  not  finishing  on 
budget  (Nelson,  Clark,  &  Spurlock,  1999). 

Summary 

The  literature  in  general  project  management  and  software  project  management  shows 
many  factors  go  into  producing  a  successful  project.  Critical  success  factors  demand  the 
attention  of  the  project  manager  while  administering  the  project  through  its  life  cycle.  Another 
issue  is  the  priority  the  project  manager  gives  each  of  the  traditional  success  criteria  and 
customer  satisfaction.  Even  the  definition  of  project  success  becomes  a  matter  of  importance:  to 
have  a  truly  satisfied  customer,  the  project  manager  must  go  beyond  considering  just  the 
schedule,  budget,  and  requirements.  Having  the  user  interact  with  the  project's  development  can 
go  a  long  ways  toward  securing  that  satisfaction.  But  little  research  has  been  done  on  Air  Force 
software  project  management  and  its  possible  differences  from  the  private  sector.  If  these 
differences  are  great,  then  private  sector  research  on  project  success  is  not  easily  generalizable  to 
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the  Air  Force.  Consequently,  this  study  sought  to  explore  this  issue  by  surveying  Air  Force 
software  project  managers  on  their  project  goals,  user  interaction  and  project  success,  and  the 
factors  they  consider  critical  for  success. 
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III.  Methodology 


Overview 

Software  project  managers  responded  to  an  online  questionnaire  consisting  of  three  parts: 
ipsative  questions  on  success  criteria  priorities,  Likert  scale  questions  on  user  involvement  and 
success,  and  open-ended  questions  to  capture  their  perceptions  of  CSFs.  All  reasonable  efforts 
were  taken  to  ensure  the  reliability  and  validity  of  the  questionnaire  items. 

Respondents 

Software  project  managers  from  Organization  Alpha  and  Organization  Bravo  responded 
to  an  online  questionnaire.  Both  Alpha  and  Bravo  develop  and  maintain  software  for  the  Air 
Force.  Both  operate  at  Level  3  on  the  Software  Development-Capability  Maturity  Model.  They 
have  documented,  standardized,  and  defined  their  work  processes  in  management  and  software 
engineering  so  that  results  are  repeatable  and  can  be  analyzed  to  some  basic  extent.  Alpha  has 
approximately  565  civilians,  178  officers,  and  678  enlisted.  Bravo  has  approximately  546 
civilians,  29  officers,  and  28  enlisted.  Most  software  project  managers  and  program  managers 
are  civilians  and  officers,  while  most  of  the  code  programmers  and  technology  experts  are 
enlisted  and  civilians. 

Commanders  of  both  organizations  endorsed  the  invitation  to  take  this  questionnaire  and 
emailed  the  invitation  to  all  of  their  software  program  managers  and  project  managers  (Appendix 
A).  The  questionnaire  was  available  online  from  December  15th  to  January  20th.  The 
commanders  sent  reminders  on  January  13th.  Stanton  (1998)  indicates  measurement  equivalence 
between  internet  questionnaires  and  traditional  paper  versions.  A  copy  of  the  questionnaire  is  in 
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Appendix  B.  The  questionnaire  refers  to  software  in  general  because  the  project  managers  dealt 
with  a  variety  of  software  products,  each  with  different  levels  of  complexity  and  source  lines  of 
code. 

Description  of  the  Questionnaire 

The  research  literature  on  CSFs  has  used  two  methodologies:  the  case  study  and  the 
questionnaire.  The  case  study  commonly  involves  the  researcher  visiting  one  organization, 
interviewing  the  project  managers,  and  drawing  conclusions  on  that  one  organization.  This  study 
used  the  questionnaire  method  to  study  multiple  organizations,  elicit  information  on  specific 
subjects,  and  infer  conclusions  on  multiple  organizations. 

The  questionnaire  addressed  the  three  investigative  questions  by  asking  about  the 
relevant  constructs:  1)  success  criteria  and  goals,  2)  user  interaction  and  satisfaction,  and  3) 
CSFs.  Because  each  construct  required  a  methodologically  different  set  of  items,  the 
questionnaire  was  divided  into  four  parts:  the  first  part  for  demographic  data  and  the  other  parts 
for  each  construct. 

To  answer  the  first  investigative  question,  an  ipsative  questionnaire  asked  respondents  to 
prioritize  project  success  criteria.  The  questionnaire  repeatedly  asked  the  respondents  to  show 
preference  for  one  of  two  success  criteria  under  different  software  projects  and  with  different 
combinations  of  criteria.  Ideas  for  the  different  software  projects  categories  came  from  examples 
described  in  Alpha’s  and  Bravo’s  websites.  These  items  were  Part  II  of  the  questionnaire. 

For  each  software  project,  the  respondents  selected  one  of  the  success  criteria  twice,  the 
second  criteria  once,  and  the  third  criteria  not  at  all.  The  success  criteria  selected  twice  is  the 
respondent's  greatest  preference,  and  the  one  selected  not  at  all  is  the  lowest  preference.  These 
results  formed  a  matrix  of  priorities  for  each  software  system  and  showed  the  level  of  preference 
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for  each  success  criteria.  The  questionnaire  used  5  different  projects  and  had  3  items  per  project 
mixed  in  the  15  items.  An  example  item  asked,  “While  developing  a  new  personnel  system, 
which  is  more  important?  on  budget  or  meeting  expectations.” 

The  second  investigative  question  was  addressed  by  having  respondents  answer  Slevin 
and  Pinto’s  (1986)  questionnaire  for  the  following  CSFs  on  user  interaction:  client  consultation, 
client  acceptance,  and  communication.  In  Slevin  and  Pinto’s  questionnaire,  user  interaction 
consists  of  the  project  manager’s  efforts  to  consult  with  the  client  on  the  project’s  progress, 
secure  the  client’s  acceptance  of  this  progress,  and  maintain  open  communication  with  the  client. 
The  respondents  also  answered  Pinto  and  Slevin’s  (1988)  questionnaire  on  project  success, 
which  was  designed  to  evaluate  the  success  of  a  project  from  the  project  manager's  perceptions. 
The  items  addressing  this  investigative  question  are  in  Part  III  of  the  questionnaire.  The  items 
for  this  part  of  the  questionnaire  were  answered  with  1 1 -point  Likert  scales.  For  example,  here  is 
an  item  for  project  success:  “This  project  has/will  come  in  on  schedule.  Strongly  Disagree  0,1, 
2,  3,  4,  5,  6,  7,  8,  9,  10  Strongly  Agree.  5  is  Neutral.” 

And  for  the  third  investigative  question,  the  respondents  selected  the  10  factors  they 
considered  most  critical  for  project  success  from  a  list  of  23  taken  from  literature  on  project 
management  (White  &  Fortune,  2002).  Items  addressing  this  question  are  in  Part  IV  of  the 
questionnaire. 

Development  and  Validation  of  the  Questionnaire 

Numerous  steps  were  taken  to  ensure  the  reliability  and  validity  of  the  questionnaire. 
Appropriate  and  valid  instruments  were  derived  from  peer-reviewed  literature  to  address  each 
construct.  These  instruments  and  methodologies  were  reviewed  in  other  literature  for 
improvements  and  critiques.  The  questionnaire  was  reviewed  in  a  pilot  test  and  pretest  by 
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students  and  professors  in  software  engineering  and  information  resource  management.  The 
research  in  progress  was  briefed  in  an  academic  conference  for  feedback.  Professors  and  the 
research  sponsor  helped  ensure  the  questionnaire’s  wording  was  understandable  to  the  intended 
audience.  And  after  the  questionnaire  was  conducted,  the  data  provided  strong  reliability  scores 
and  factor  loading. 

1 .  Derived  instruments  from  literature:  Each  part  of  the  questionnaire  used  a  different 
instrument  methodology  to  properly  address  the  different  constructs. 

a.  Tamir  and  Lunetta’s  (1977)  research  indicates  the  ipsative  method  (also  known  as 
paired  comparison)  is  an  appropriate  way  to  survey  preferences.  Relevant  items 
were  made  under  this  methodology. 

b.  Slevin  and  Pinto  (1986)  and  Pinto  and  Slevin  (1988)  provide  validated  items  for 
the  constructs  of  user  interaction  and  project  success.  These  Likert  scale  items 
were  useful  in  learning  the  degree  each  respondent  supported  user  interaction  and 
the  extent  of  his/her  project’s  success.  Pinto  and  Slevin  also  developed  standards 
to  gauge  whether  a  project  was  successful,  approaching  success,  or  unsuccessful. 
These  standards  were  based  on  their  study  of  418  projects  (Slevin  &  Pinto,  1988). 

c.  White  and  Fortune  (2002)  describe  their  questionnaire  in  detail,  one  part  of  which 
dealt  with  CSFs.  To  best  compile  CSF  examples,  respondents  had  to  have  a 
choice  over  what  they  selected,  a  list  to  expedite  their  search  for  words  to  describe 
these  CSFs,  and  a  blank  box  so  they  could  add  in  any  perceived  CSFs  not  on  the 
list.  White  and  Fortune’s  items  were  also  useful  because  they  provided  raw  data 
to  compare  results  with,  their  research  was  recent,  and  their  list  of  potential  CSFs 
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was  consistent  with  the  rest  of  the  questionnaire  by  including  the  CSFs  found  by 
Slevin  and  Pinto  (1986). 

2.  Reviewed  literature’s  validation  of  instruments: 

a.  Tamir  and  Lunetta’s  (1977)  research  indicates  ipsative  procedures  with 
approximately  20  items  have  satisfactory  internal  consistency  coefficients. 

b.  Slevin  and  Pinto  (1986)  designed  and  validated  their  diagnostic  survey  instrument 
with  psychometric  properties  to  assign  scores  to  the  CSFs  of  a  project.  The 
instrument  has  5  items  for  each  of  the  10  CSFs.  Slevin  and  Pinto  conducted  a 
corrected  item-total  correlation  to  find  good  questions  that  strongly  correlated 
with  the  results  from  the  other  questions.  This  technique  provided  internal 
consistency  for  the  scale  items.  When  Slevin  and  Pinto  surveyed  85  project 
managers,  the  questionnaire  yielded  a  Cronbach  alpha  coefficient  averaging  0.86. 
Pinto  and  Slevin  (1988)  added  a  section  to  their  diagnostic  instrument  to  measure 
a  project  manager’s  perception  of  a  project’s  success.  They  validated  the 
questionnaire  on  CSFs  and  project  success  based  off  of  a  study  of  418  projects. 

c.  The  Likert  scales  for  items  on  client  consultation,  client  acceptance, 
communication,  and  feedback  are  1 1 -point  scales.  Hinkin  (1998)  notes 
coefficient  alpha  reliability  for  Likert  scales  increase  up  to  five  points,  but  do  not 
gain  much  more  value  with  more  points.  The  1 1 -point  scales  helped  provide 
variance  among  answers. 

d.  White  and  Fortune  (2002)  derived  the  23  factor  list  from  an  exhaustive  review  of 
CSF  project  management  literature.  They  also  include  a  24th  blank  factor  so 
respondents  can  provide  additional  factors  they  consider  critical  to  success. 
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3.  Read  critiques  of  the  questionnaires:  Slevin  and  Pinto  (1986),  Pinto  and  Slevin  (1988), 
and  Pinto  and  Prescott  (1988)  tested  their  questionnaire  every  time  they  published  new 
research  on  CSFs  and  project  success.  Belout  (1998)  also  critiqued  the  questionnaire. 
Overall,  the  items  were  found  to  be  valid  and  reliable  for  the  study  on  project 
management. 

a.  Pinto  and  Prescott  (1988)  tested  the  reliability  and  validity  of  the  ten  CSFs  by 
analyzing  answers  from  408  project  manager  respondents  who  took  the 
questionnaire.  Just  as  before  (Slevin  &  Pinto,  1986),  the  Cronbach  alpha  scores 
for  the  internal  consistency  of  the  CSFs  were  high,  ranging  from  0.79  to  0.90. 
Pinto  and  Prescott  conducted  a  confirmatory  factor  analysis  for  each  of  the  CSFs, 
which  indicated  construct  validity  for  every  factor.  “The  factor  loadings  ranged 
from  0.49  to  0.90  with  the  average  being  0.64"  (Pinto  &  Prescott,  1988,  p.  1 1). 

b.  In  1990,  Pinto  and  Mantel  use  the  CSF  questionnaire  again,  this  time  to  study 
project  failure  in  R&D  and  construction.  They  hypothesized  that  if  CSFs  lead  to 
project  success,  then  their  absence  should  lead  toward  project  failure.  Pinto  and 
Mantel  surveyed  130  people,  mostly  from  the  Project  Management  Institute,  a 
professional  association  of  project  managers.  97  people  responded,  which  yielded 
a  75%  response  rate.  T-tests  showed  no  bias  on  research  variables  for  early  and 
late  respondents. 

c.  Pinto  and  Mantel  (1990)  tested  the  CSF  construct  again  for  internal  consistency, 
and  they  once  again  derived  Cronbach  alpha  scores  ranging  from  0.79  to  0.90. 
They  conducted  a  confirmatory  factor  analysis  on  the  dependent  variables  of 
project  failure/success  to  determine  whether  “[they]  were,  in  fact,  a  valid 
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subdimensional  representation  of  the  elements  of  perceived  success  or  failure  of  a 
project”  (Pinto  &  Mantel,  1990,  p.  272).  The  failure/success  construct  did  emerge 
and  accounted  for  66.9%  of  total  variance  in  project  failure. 

d.  Pinto  and  Mantel  (1990)  used  stepwise  regression  to  analyze  how  strongly  each 
absent  CSF  impacted  different  projects  at  different  stages  under  different  failure 
criteria.  Their  results  confirm  earlier  work:  the  CSFs  were  definitely  critical  in 
avoiding  project  failure.  Thus,  CSFs  for  success  and  their  absence  for  failure 
strongly  correlate  and  indicate  convergent  validity. 

e.  Belout  (1998)  argues  that  Pinto  and  Slevin  do  not  rigorously  define  the  dependent 
variable  "project  success"  nor  precisely  measure  it.  He  considers  Pinto  and 
Slevin's  literature  review  of  project  success  to  be  a  limited  validation  of  the 
measure.  "This  is  very  critical  since  a  debate  exists  about  the  complexity  of  this 
construct  [project  success]"  (Belout,  1998,  p.  12). 

f.  There  were  no  critiques  available  for  White  and  Fortune's  study  (2002)  because  it 
was  recently  published. 

4.  Pilot  test:  To  improve  this  survey,  a  pilot  test  was  conducted  to  identify  any  poorly 

worded  items  and  practice  analyzing  data  to  make  sure  it  revealed  the  useful  information. 
Twelve  respondents  from  the  Air  Force  Institute  of  Technology  (AFIT)  critiqued  the  pilot 
test:  seven  software  engineering  graduate  students  enrolled  in  an  advanced  software 
engineering  class  and  five  software  engineering  professors.  This  group  had 
representative  characteristics  of  the  population  for  this  study  because  of  their  educational 
and  occupational  background.  They  had  an  average  of  6.4  years  of  software  project 
management  experience.  They  recommended  standardizing  the  questionnaire's 
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terminology  with  words  and  meanings  that  software  engineers  and  project  managers 
commonly  use  and  have  a  common  understanding. 

5.  Pre-test:  One  professor  and  ten  graduate  students  in  AFIT's  information  resource 
management  program  tested  the  online  version  of  the  questionnaire  to  critique  the 
formatting,  spelling,  and  wording,  and  to  ensure  the  questionnaire  detected  improper 
answers,  recorded  all  answers,  and  was  user  friendly.  For  example,  the  respondents 
recommended  putting  a  point  of  contact  on  the  questionnaire  in  case  respondents  had 
questions.  They  also  recommended  offering  an  executive  summary  of  the  research 
results  to  respondents.  This  would  act  as  an  incentive  for  anyone  curious  about  the 
research  and  they  would  see  that  their  input  was  used. 

6.  Briefed  at  conference:  This  research  was  briefed  for  peer  review  during  an  academic 
conference.  Ten  business  and  project  management  professors  critiqued  the  methodology 
and  literature  review. 

7.  Reviewed  with  software  project  manager  and  used  standardized  terminology: 

a.  A  software  project  manager  consultant  in  Organization  Alpha  reviewed  multiple 
versions  of  the  questionnaire  and  recommended  improvements  in  terminology  so 
it  would  make  sense  to  the  prospective  respondents.  While  her  suggestions 
changed  some  wording  and  added  clarifications,  the  content  of  the  questionnaire 
remained  true  to  the  literature  it  was  derived  from.  For  example,  where  Slevin 
and  Pinto  (1986)  refer  to  "clients"  who  are  paying  for  the  project,  the 
questionnaire  now  refers  to  "customers"  to  fit  with  the  terminology  in  the  Systems 
Engineering  Process  List  of  Definitions  (Anon,  2002).  This  list  of  definitions  and 
the  consultant’s  proofreading  helped  clarify  the  distinctions  between  similar 
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words  and  ensure  the  correct  interpretation  of  each  item’s  intended  meaning.  For 
example,  while  a  “customer”  is  the  person  or  organization  paying  for  the  software 
product,  “users”  are  the  people  who  will  use  the  software  product  in  their  daily 
work. 

b.  Terminology  was  also  taken  from  an  online  software  professional  development 
curriculum  that  was  designed  for  software  project  managers  at  Alpha,  Bravo,  and 
any  other  Air  Force  software  developing  units.  For  example,  the  questionnaire 
asks  the  software  project  managers  what  phase  of  the  software  lifecycle  their 
project  is  currently  going  through.  Because  literature  offers  a  number  of  software 
lifecycle  models  (like  the  Waterfall  Model,  the  Sawtooth  Model,  and  the  Spiral 
Model),  examples  of  life  cycle  phases  came  from  the  software  professional 
development  curriculum.  This  ensured  the  respondents  had  a  common  frame  of 
reference  for  software  project  management  concepts. 

c.  The  researcher  also  visited  the  headquarters  of  Organization  Alpha  to  brief 
prospective  respondents  on  the  general  problem  background  of  the  research  and 
elicit  their  feedback. 

8.  Checked  reliability  and  factor  analysis:  The  Likert  scale  items  in  Part  III  of  the 
questionnaire  was  appropriate  for  factor  analysis.  The  items  referring  to  user 
consultation,  user  commitment,  communication,  and  project  success  were  analyzed  with 
factor  analysis.  The  set  of  items  for  each  construct  loaded  strongly  on  one  component. 
Within  a  rotated  component  matrix,  most  items  loaded  above  0.8  in  the  varimax  rotation 
method.  Under  Cronbach  reliability  analysis,  all  sets  of  items  had  at  least  a  Cronbach 
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alpha  of  0.8015  up  to  0.9566.  A  Cronbach  alpha  of  at  least  0.70  indicates  satisfactory 
reliability  (Nunnally,  1976). 

Methodology  for  Data  Analysis 

For  the  first  investigative  question,  the  data  was  analyzed  by  simply  taking  the  statistical 
mean  of  all  respondents'  preferences.  This  provided  the  preferences  of  software  project 
managers  in  prioritizing  success  criteria  by  showing  how  each  criterion  was  prioritized  first  on 
average.  This  observation  indicated  software  project  managers’  priorities  for  meeting 
requirements,  schedules,  and  budgets.  For  example,  if  50  project  managers  prioritized  between 
budget  and  schedule,  and  45  of  them  prioritized  budget  first,  then  the  budget  became  a  higher 
priority  on  average.  This  indication  becomes  more  accurate  as  the  project  managers  repeatedly 
must  make  priority  decisions  throughout  this  part  of  the  questionnaire. 

For  the  second  investigative  question,  Pinto  and  Sievin’ s  (1988)  questionnaire  was 
designed  to  convert  data  on  user  interaction  and  project  success  into  scores  to  measure  the 
success  of  the  project  under  user  satisfaction  and  the  traditional  success  criteria  (requirements, 
budget,  and  schedule).  By  studying  418  projects,  Pinto  and  Slevin  developed  standardized 
scores  of  project  success  that  classified  a  project's  status  as  good  (above  the  80th  percentile  of 
the  success  scores  the  418  project  achieved),  fair  (between  the  50th  and  80th  percentile),  and 
critical  (below  the  50th  percentile).  A  good  project  was  successful  in  practically  all  ways  (on 
time,  on  budget,  fulfills  requirements,  and  satisfies  user).  A  fair  project  fulfilled  some  or  most 
criteria  for  success.  And  a  critical  project  failed  most  to  achieve  most  criteria  and  required 
serious  attention  to  fix.  Client  consultation,  client  acceptance,  and  communication  are 
considered  independent  variables  because  Pinto  and  Mantel  (1990)  had  already  tested  and  found 
a  cause-and-effect  relationship  between  the  10  CSFs  and  the  dependent  variables,  which  are 
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project  success  and  client  satisfaction.  Multiple  regression  in  this  study  was  used  to  determine 
how  strongly  the  independent  variables  of  user  interaction  correlated  to  the  dependent  variable  of 
project  success 

And  for  the  third  investigative  question,  the  nonparametric  data  was  examined  with 
Kendall’s  and  Spearman’s  Rank  Correlation  methods.  Each  respondent  selected  10  CSFs  out  of 
a  list  of  23  possible.  A  rank  ordering  of  CSFs  was  derived  from  adding  up  how  many  times  each 
CSF  was  selected.  Kendall’s  and  Spearman’s  Rank  Correlation  methods  were  used  to  detect 
any  statistically  significant  differences  between  the  rank  orders  of  this  study's  results  and  White 
and  Fortune's  (2002).  These  methods  use  a  null  hypothesis  and  an  alternate  hypothesis  in  a  two- 
tailed  test: 

Hn:  r  =  0. 

Ha:  r^O. 

Spearman’s  rank  correlation  coefficient,  r  (Spearman's  rho),  indicates  how  well  the  two 
rank  orderings  correlate.  A  perfect  positive  correlation  would  be  r  =  1 .  A  perfect  negative 
correlation  would  be  r=  -1.  As  the  coefficient  of  r  approaches  0,  the  correlation  decreases  to 
the  point  of  0,  in  which  there  is  absolutely  no  correlation. 

Kendall’s  method  is  intended  for  a  smaller  number  of  responses  (fewer  than  10). 

Kendall's  method  thus  works  well  for  deeper  analysis  of  the  data  for  demographic  groupings 
within  the  larger  respondent  population  of  64  (e.g.:  there  were  9  project  managers  in  charge  of 
projects  developed  by  contractors). 

By  contrasting  the  results  of  this  study  with  White  and  Fortune's  results,  this  study 
identified  differences  between  Air  Force  software  project  managers  perceived  CSFs  and  the 
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private  sector.  Comparisons  were  also  made  between  cross  sections  of  the  data  for  demographic 
groupings. 

By  comparing  data  within  demographic  groupings,  this  study  sought  to  infer  more  than 
the  cumulative  average.  These  demographic  groupings  break  respondents  up  by  service  (military 
and  civil  service),  project  role  (project  manager  and  program  manager),  veteran  experience  in 
software  engineering  and/or  project  management  (  10  >  x  >  5  years  and  x  >  15  years),  type  of 
project  (sustaining  a  legacy  system  and  developing  a  new  product),  primary  developer 
(government,  commercial  off  the  shelf,  and  contractor),  and  the  project's  status  (good  and 
critical). 

The  military  service  members  include  both  enlistees  and  officers.  Although  enlistees 
usually  have  responsibilities  different  from  officers,  the  two  enlisted  respondents  classified 
themselves  as  software  project  managers,  rather  than  as  project  leads,  project  team  members,  or 
other.  Therefore,  the  enlistees  probably  had  the  same  duties  and  software  project  management 
knowledge  as  the  officer  respondents. 

White  and  Fortune  (2002)  include  past  experience  as  one  of  the  CSFs  that  other 
researchers  had  identified.  Therefore,  project  manager  experience  was  used  as  a  discriminator  to 
see  how  it  affected  the  other  variables.  5  years  is  the  baseline  because  the  Project  Management 
Institute  requires  a  person  (among  other  criteria)  to  have  at  least  3  years  of  experience  managing 
a  project  to  apply  for  the  association's  certification  as  a  project  manager.  As  such,  5  years  was  a 
conservative  baseline  to  expect  project  managers  to  make  good  decisions  in  project  management 
through  lessons  learned  from  experience.  The  5  to  10  year  group  of  respondents  is  contrasted 
with  the  1 5+  year  respondents  to  look  for  any  conspicuous  differences  between  the  two  that 
might  not  be  visible  by  also  including  the  10  to  15  year  group. 
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Summary 

This  exploratory  study  sought  to  examine  many  different  topics,  each  which  required  a 
different  questionnaire  and  methodology  that  was  best  suited  to  it  (Table  4).  This  study  collected 
reliable  and  valid  data  to  analyze  and  learn  about  Air  Force  software  project  management. 


Table  4 

Summary  of  Methodology^ 


Investigative  Question 

Questionnaire  Measures 

Analysis 

Methodology 

1 

How  do  Air  Force  software  project 
managers  prioritize  the  three  traditional 
measures  of  project  success? 

15  items,  ipsative  (Tamir& 
Lunetta,  1977) 

Statistical  Means 

2 

Does  high  user  interaction  in  Air  Force 
software  projects  correlate  with  project 
success? 

27  items,  1 1 -point  Likert 
scales  (Pinto  &  Slevin,  1988; 
Slevin  &  Pinto,  1986) 

Multiple  Regression 

3 

Are  the  critical  success  factors  for  Air 
Force  software  projects  different  from 
private  sector  projects? 

23  items,  open-ended  (White 
&  Fortune,  2002) 

Spearman's  Rank 
Correlation 
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IV.  Data  Analysis 


Overview 

The  respondents  provided  a  variety  of  data  for  the  questionnaires.  A  straightforward  look 
at  the  data  indicates  answers  to  this  exploratory  study's  investigative  questions  on  goals,  user 
interaction  and  project  success,  and  CSFs.  However,  even  more  can  be  inferred  by  examining 
the  data  through  cross  sections  of  the  respondents. 

Demographics 

The  questionnaire  invitation  was  sent  to  214  people  whom  Organization  Alpha  and 
Organization  Bravo  classified  as  software  project  managers.  There  were  71  respondents,  which 
yielded  a  33.2%  response  rate.  Six  respondents  only  answered  half  the  questionnaire,  and  they 
are  consequently  not  included  in  the  data  analysis  for  the  second  and  third  investigative 
questions.  Likewise,  one  respondent  invalidated  his  answers  by  stating  he  was  not 
knowledgeable  enough  to  properly  answer  the  questions.  Raw  data  on  demographics  is  available 
in  Appendix  C. 
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The  majority  of  the 


Table  5 


respondents  were  civil 
servants  (Table  5),  with  an 
average  of  24.5  years  of 
time  in  government  service 
and  12  years  of  experience 
with  project  management 
and/or  software 
engineering. 

Approximately  half  of  the 
respondents  were  project 


Respondent  Demographics,  by  Pay  Grade 


Service  Category 

and  Pay  Grade 

Number  of 

Respondents 

Years 

Working 

for  AF 

Experience  with 

Project  Management 

and/or  Software 

Engineering  (Years) 

GS-10  to  GS-14 

55 

24.5 

12 

0-4  to  0-6 

1 

6 

3 

0-1  to  0-3 

11 

7.7 

1.5 

E-7  to  E-9 

2 

21 

6 

Other 

1 

28 

22 

managers  (in  charge  of  one  project)  and  the  other  half  were  program  managers  (in  charge  of 
multiple  projects)  (Table  6). 


Table  6 

Respondent  Demographics,  by  Project  Role 


# 

Role 

# 

Service  Category 
and  Pay  Grade 

Years  Working 
For  AF 

Years  of  Experience 
in  Project 

Management  and/or 
Software  Engineering 

28 

program  manager 

21 

GS-10  to  GS-14 

21.5 

11.8 

6 

0-1  to  0-3 

10.8 

2 

1 

other 

28 

22 

24 

project  manager 

16 

GS-10  to  GS-14 

27.1 

14.1 

1 

0-4  to  0-6 

6 

3 

5 

0-1  to  0-3 

4 

0.8 

2 

E-7  to  E-9 

21 

6 

9 

project  lead 

9 

GS-10  to  GS-14 

23.1 

14.3 

3 

project  team  member 

3 

GS-10  to  GS-14 

25.7 

9.3 

6 

other 

6 

GS-10  to  GS-14 

29.3 

5.3 
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First  Investigative  Question 

1 .  How  do  Air  Force  software  project  managers  prioritize  the  three  traditional  measures  of 
project  success? 

Respondents  prioritized  "meeting  expectations"  first  56.6%  of  the  time,  distantly 
followed  by  "on  time"  (24.2%)  and  "on  budget"  (19.2%)  (Appendix  D).  Table  7  shows  how 
other  cross  sections  of  the  respondents  prioritized  their  goals. 


Table  7 

Goals  by  Cross  Section 


Selected  %  of  the  Responses 

Cross  Section 

Time 

Budget 

Meeting 

Expectations 

Average  of  All  Respondents 

24.2 

19.2 

56.6 

Project  Status 

Good 

23.9 

22.1 

53.9 

Critical 

22.5 

19.7 

57.8 

Experience  Software 
Engineering  and/or 

£  15  Years 

26.7 

15.2 

58.1 

Project  Management 

10  >  x  >  5  Years 

20.0 

18.7 

61.3 

Project  Type 

Sustaining 

25.3 

19.4 

55.2 

New  Developing 

20.0 

22.0 

58.0 

Service 

Military 

23.8 

26.7 

49.5 

Civil  Service 

24.6 

17.1 

58.3 

Developer 

GOTS 

26.2 

18.7 

55.1 

COTS 

15.2 

28.6 

56.2 

Contractor 

20.7 

15.6 

63.7 

Role 

Project  Manager 

27.2 

17.8 

55.0 

Program  Manager 

27.2 

17.8 

55.0 

Note.  Military  service  respondents  included  officers  and  enlistees. 


Table  7  shows  that  most  of  the  cross  sections  closely  follow  the  average  of  all 
respondents.  The  military  members  are  slightly  different  with  a  higher  concern  for  budgets,  and 
COTS  project  managers  are  the  most  concerned  about  the  budget  and  least  concerned  about  the 
schedule  out  of  all  the  respondents,  and  the  project  managers  for  contracted  projects  are  the  most 
concerned  about  meeting  expectations  and  the  least  concerned  about  finishing  on  budget. 
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Second  Investigative  Question 

2.  Does  high  user  interaction  in  Air  Force  software  projects  correlate  with  project  success? 
Multiple  regression  analysis  of  the  data  indicates  an  adjusted  r  square  of  0.735,  which 

suggests  the  CSFs  for  user  interaction  account  for  a  large  part  of  project  success.  The  multiple 
regression  model  was  also  statistically  significant  ( p  <  0.05).  Most  of  the  64  surveyed  projects 
were  sustaining  a  legacy  system  (35)  versus  a  new  start  software  development  effort  (17),  and 
most  projects  were  government  developed  (41)  rather  than  based  off  of  a  contracted  project  (9) 
or  commercial  off  the  shelf  software  (7)  (Appendix  E).  The  respondents  provided  a  diversity  of 
projects  for  analysis,  rather  than  solely  claiming  all  projects  had  high  levels  of  success.  22  were 
good  (successful),  1 8  were  fair  (approaching  success),  and  24  were  critical  (unsuccessful) 
(Appendices  F  and  G).  Appendix  H  shows  on  average  that  items  were  answered  in  the  positive 
half  of  the  Likert  scale,  but  the  standard  deviations  reveal  there  was  a  wide  range  of  responses  to 
some  of  the  items. 

The  project  managers  with  5  to  10  years  of  experience  had  the  highest  percentage  of 
failed  projects  for  their  experience  group  (69%  were  critical)  and  the  fewest  successful  projects 
(15%  were  good).  This  is  surprising  because  this  group  actually  had  a  lower  perception  of  their 
project  success  than  the  group  with  less  than  5  years  of  experience  (42%  were  critical  and  26% 
were  good).  The  group  of  project  managers  with  15+  years  of  experience  had  the  highest 
success  rate  (53%  were  good)  and  the  lowest  rate  of  failure  (16%  were  critical). 

Third  Investigative  Question 

3.  Are  the  critical  success  factors  for  Air  Force  software  projects  different  from  private 
sector  projects? 
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The  rank  comparison  between  the  research  results  and  White  and  Fortune's  data  (2002) 
using  Kendall's  tau  correlation  and  the  Spearman  rank  order  correlation  revealed  a  positive 
correlation  between  the  pair  of  rank  orderings  (Table  8).  The  model  was  statistically  significant. 


However,  Spearman’s  correlation 

Table  8 

coefficient  of  0.547  indicates  the 

correlation  is  not  perfect  and  that  the 

Rank  Comparison  between  Research  Results  and  White 

and  Fortune  (2002) 

surveyed  project  managers  had  other 

Method 

Correlation  Coefficient 

CSFs  than  the  private  sector.  The 

Kendall’s  tau-b 

0.436 

public  and  private  sector  agreed  on 

Spearman's  rho 

0.547 

the  top  seven  CSFs,  but  not  in  the 

Note.p  <  0.05 

same  order  (Table  9). 

The  sharp  difference  in  lower  ranked  factors  is  most  notable  in  the  CSFs  ranked  6,  8.5 
(had  a  tie),  and  1 1  by  the  Air  Force  software  project  managers.  The  private  sector  project 
managers  correspondingly  ranked  them  20,  21.5,  and  23.  The  percent  of  project  managers 
selecting  each  CSF  emphasized  the  distinct  difference  in  choice  between  private  and  public 
sector  respondents.  For  example,  for  the  CSF  of  “support  from  stakeholders  and  champions,” 
53.1%  of  Air  Force  project  managers  selected  it,  while  only  1.3%  of  private  sector  project 
managers  selected  it.  Table  9  displays  a  complete  list  of  rankings  between  this  study’s  results 
and  the  results  of  White  and  Fortune  (2002). 

Respondents  offered  four  more  CSFs  that  they  recommended  adding  to  the  list  of  23 
(Appendix  I). 
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Table  9 

Rank  Comparison  Between  CSFs,  Research  Results  and  White  and  Fortune  ( 2002 ) 


Rank 

%  of  Respondents 
Who  Selected  this 
CSF 

CSFs 

Research 

Results 

White  & 
Fortune, 
2002 

Research 

Results 

White  & 
Fortune, 
2002 

Adequate  funds/resources 

1 

4 

93.8 

69.5 

Clear  goals/objectives 

2 

1 

87.5 

87.3 

Realistic  schedule 

3 

2 

85.9 

78.4 

End  user  commitment 

4 

5 

79.7 

67.4 

Clear  communication  channels 

Having  access  to  innovative/talented 

5 

6 

71.9 

61.0 

people 

6 

20 

64.1 

3.4 

Support  from  senior  management 

7 

3 

54.7 

74.6 

Effective  management  of  risk 

8.5 

13.5 

53.1 

49.6 

Support  from  stakeholder(s)/champion(s) 

8.5 

21.5 

53.1 

1.3 

Effective  leadership/conflict  resolution 

10 

7 

50.0 

58.5 

Having  a  clear  project  boundary 

11 

23 

40.6 

0.8 

Effective  team  building/motivation 

13.5 

13.5 

34.4 

49.6 

Flexible  approach  to  change 

13.5 

9 

34.4 

56.4 

Having  relevant  past  experience 

13.5 

21.5 

34.4 

1.3 

Recognizing  complexity 

13.5 

10.5 

34.4 

51.3 

Effective  monitoring  and  feedback 

16 

8 

29.7 

57.2 

Training  provision 

17 

15 

25.0 

41.5 

Taking  account  of  past  experience 

18 

10.5 

18.8 

51.3 

Taking  account  of  external  influences 

19 

12 

15.6 

50.8 

Considering  multiple  views  of  project 

20.5 

19 

9.4 

19.9 

Provision  of  planning  and  control  systems 

20.5 

17 

9.4 

37.3 

Contextual  awareness 

22 

16 

7.8 

39.8 

Appreciating  the  effect  of  human  error 

23 

18 

6.3 

22.5 

The  demographic  groups  had  very  similar  rankings  to  the  average  (Table  10).  The 
groups  that  were  compared  with  each  other  strongly  agreed  on  the  ranking.  Surprisingly,  the 
Good  Projects  and  Critical  Projects  had  a  very  high  Spearman's  rho  of  0.91.  This  indicates  the 
software  project  managers  for  both  successful  and  unsuccessful  projects  agree  on  CSFs,  but  the 
successful  project  managers  are  better  at  applying  those  CSFs.  Likewise,  the  newer  project 
managers  (5  up  to  10  years  of  experience)  and  more  experienced  ones  (15+  years)  strongly 
agreed  on  CSFs  (0.94)  but  the  newer  project  managers  still  had  a  much  higher  failure  rate. 
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Table  10 

Rank  Correlation  Amonc[  Groups 


Compared  Rankings 

Spearman's  rho 

Kendall's  tau-b 

Research 

White  &  Fortune,  2002 

0.55 

0.44 

5  <  x  <  10  Years 

15  <  x  Years 

0.94 

0.82 

Good  Project 

Critical  Project 

0.91 

0.77 

Project  Manager 

Program  Manager 

0.86 

0.72 

Military 

Civil  Service 

0.84 

0.70 

Sustainment 

New  Development 

0.81 

0.61 

GOTS 

COTS 

0.85 

0.70 

GOTS 

Contractor  Developed 

0.86 

0.70 

COTS 

Contractor  Developed 

0.66 

0.54 

Note.  GOTS  stands  for  government  developed.  COTS  stands  for  commercial 
off  the  shelf,  p  <  0.05 


Despite  the  strong  agreement  in  ordering  of  CSFs,  there  were  still  conspicuous 
differences  between  demographic  groups  that  could  be  studied  with  further  research.  A  full 
comparison  of  CSFs  between  demographic  groups  is  in  Appendices  J,  K,  L,  M,  N,  and  O. 

For  example,  in  comparison  with  GOTS  and  COTS  (Table  1 1),  far  fewer  respondents  for 
contracted  software  projects  valued  effective  management  of  risk  (33%),  support  from  senior 
management  (44%),  and  training  provisions  (0%).  Perhaps  the  project  managers  for  contracted 
projects  consider  risk  management  and  training  provisions  to  be  the  responsibility  of  the 
contracted  company,  and  senior  management  is  already  committed  to  the  projects  because  of 
contracts.  On  the  other  hand,  in  comparison  with  GOTS  and  contracted  projects,  far  fewer 
COTS  respondents  valued  having  a  clear  project  boundary  (0%),  recognizing  complexity  (14%), 
and  having  relevant  past  experience  (14%).  The  COTS  project  managers  may  have  this  view 
because  commercial  off  the  shelf  products  are  by  their  very  nature  supposed  to  be  simpler  than  a 
product  that  must  be  developed  from  the  ground  up  and  the  project  boundary  should  be  clear  cut. 
Strong  documentation  in  COTS  projects  may  also  lessen  the  necessity  of  past  experience  with 
similar  products. 
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Table  1 1 

CSF  Comparison  Betweer±  GOTS,  COTS,  anc[  Contractors 


CSFs 

GOTS 

Rank 

COTS 

Contractor 

%  of  Respondents 

GOTS  COTS  Contractor 

Effective  management  of  risk 

7.5 

7.5 

13.5 

56 

71 

33 

Support  from  senior  management 

7.5 

4.5 

10.5 

56 

85 

44 

Having  a  clear  project  boundary 

11 

22 

8 

39 

0 

56 

Recognizing  complexity 

12.5 

16.5 

13.5 

37 

14 

33 

Having  relevant  past  experience 

16 

16.5 

5.5 

29 

14 

67 

Training  provision 

17 

11 

22 

24 

43 

0 

Note.  CSFs  are  ordered  for  GOTS. 


There  were  also  some  distinct  differences  between  project  managers  and  program 
managers  (Table  12).  The  most  noteworthy  difference  was  in  having  relevant  past  experience. 
While  39%  of  project  managers  valued  it  as  a  top  CSF,  only  8%  of  the  program  managers  valued 
it. 


Table  12 

CSF  Comparison  Between  Project,  Managers  anc[  Program  Managers 


CSFs 

Rank 

Project  Program 

%of 

Respondents 
Project  Program 

Raw  Data,  #  of 
Respondents 
Selecting  this 
CSF 

Project  Program 

Recognizing  complexity 

10.5 

14 

48 

27 

11 

7 

Support  from  senior  management 

10.5 

5.5 

48 

69 

11 

18 

Effective  monitoring  and  feedback 

13 

18.5 

39 

15 

9 

4 

Having  relevant  past  experience 

13 

20.5 

39 

8 

9 

2 

Training  provision 

13 

16 

39 

23 

9 

6 

Considering  multiple  views  of  project 

19 

20.5 

17 

8 

4 

2 

Taking  account  of  external  influences 
Provision  of  planning  and  control 

20.5 

16 

9 

23 

2 

6 

systems 

22.5 

18.5 

4 

15 

1 

4 

Note.  Highlighted  numbers  are  the  top  10  CSFs  for  each  column.  Bold-faced  numbers  have 
conspicuous  differences.  CSFs  are  ordered  for  project  managers. 
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Three  times  as  many  military  members  are  concerned  about  external  influences  than  civil 
service  employees,  who,  in  turn,  are  three  times  as  concerned  about  building  an  effective  team 
(Table  13). 


Table  13 

CSF  Comparison  Between  Military  and  Civil  Service 


CSFs 

Rank 

Civil 

Military  Service 

%of 

Respondents 

Civil 

Military  Service 

Raw  Data,  #  of 
Respondents 
Selecting  this 
CSF 

Civil 

Military  Service 

Recognizing  complexity 

Support  from  stakeholder(s)/ 

8.5 

16 

50% 

31% 

7 

14 

champion(s) 

11.5 

7 

43% 

60% 

6 

27 

Training  provision 

11.5 

17 

43% 

22% 

6 

10 

Taking  account  of  past  experience 

14 

18 

36% 

16% 

5 

7 

Taking  account  of  external  influences 

15.5 

20.5 

29% 

11% 

4 

5 

Having  a  clear  project  boundary 

17.5 

11 

21% 

49% 

3 

22 

Having  relevant  past  experience 

17.5 

13 

21% 

42% 

3 

19 

Effective  team  building/motivation 
Provision  of  planning  and  control 

19.5 

12 

14% 

44% 

2 

20 

systems 

23 

19 

0% 

13% 

0 

6 

Note.  Highlighted  numbers  are  the  top  10  CSFs  for  each  col 
conspicuous  differences.  CSFs  are  ordered  for  military  merr 

lumn.  Bold-faced  numbers  have 
ibers. 
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Differences  also  materialized  between  project  managers  responsible  for  sustaining  a 
legacy  system  (software  maintenance)  and  project  managers  developing  new  software  (Table 
14).  29%  of  the  new  start  respondents  valued  taking  account  of  past  experience  and  taking 
account  of  external  influences,  while  11%  of  the  respondents  for  sustainment  valued  past 
experience  and  only  3%  accounted  for  external  influences  as  a  CSF. 


Table  14 

CSF  Comparison  Between  Sustaining  A  Legacy  System  and  New  Start  Software  Development 


CSFs 

Rank 

Sustain 

New 

Start 

%  of  Respondents 
New 

Sustain  Start 

Raw  Data,  #  of 
Respondents 
Selecting  this  CSF 
New 

Sustain  Start 

Effective  management  of  risk 

7 

10.5 

63 

41 

22 

7 

Having  a  clear  project  boundary 
Effective  leadership/conflict 

9.5 

19.5 

49 

18 

17 

3 

resolution 

11 

5.5 

46 

65 

16 

11 

Effective  team  building/motivation 

12 

17.5 

43 

24 

15 

4 

Having  relevant  past  experience 

13 

17.5 

40 

24 

14 

4 

Taking  account  of  past  experience 
Taking  account  of  external 

18 

14 

11 

29 

4 

5 

influences 

Appreciating  the  effect  of  human 

22 

14 

3 

29 

1 

5 

error 

23 

19.5 

0 

18 

0 

3 

Note.  CSFs  are  ordered  for  sustaining  a  legacy  system. 
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As  noted  earlier,  the  project  managers  with  5  to  10  years  of  experience  had  the  highest 
percentage  of  failed  projects  out  of  all  the  5  year  groups,  while  the  project  managers  with  15  and 
more  years  of  experience  had  the  highest  percentage  of  successful  projects.  Surprisingly,  a 
comparison  between  these  two  groups  reveals  a  very  strong  similarity  in  the  percentage  of 
respondents  choosing  CSFs  (Table  15).  The  differences  between  these  two  groups  appear  to  be 
insignificant.  The  one  CSF  they  differ  on  is  training  provisions,  with  32%  of  respondents  of  15+ 
years  choosing  it,  and  only  8%  of  project  managers  with  5  up  to  10  years  of  experience  choosing 
it. 


This  strong  agreement  in  CSFs  possibly  indicates  both  groups  of  project  managers  have 
had  the  same  project  management  and  software  engineering  education  during  the  last  five  years. 
In  which  case,  the  15+  year  group  could  be  making  wiser  use  of  the  CSFs  in  contrast  to  the  less 
experienced  group. 


Table  15 

CSF  Comparison  Between  Project  Managers  with  5  to  10  anc[  15+  Years  ofExperience 


CSFs 

Rank 

5<yrs<  10 

1 5+  yrs 

%  of  Respondents 
15+ 

5  <  yrs  <  1 0  yrs 

Raw  Data,  #  of 
Respondents 
Selecting  this  CSF 
15+ 

5  <  yrs  <  1 0  yrs 

Clear  goals/objectives 

1.5 

2.5 

100 

84 

13 

16 

End  user  commitment 

1.5 

4.5 

100 

74 

13 

14 

Realistic  schedule 

3 

2.5 

92 

84 

12 

16 

Adequate  funds/resources 

4 

1 

85 

100 

11 

19 

Having  access  to 
innovative/talented  people 

5 

4.5 

77 

74 

10 

14 

Clear  communication 
channels 

7 

6 

62 

68 

8 

13 

Effective  leadership/conflict 
resolution 

7 

7.5 

62 

58 

8 

11 

Support  from 

stakeholder(s)/champion(s) 

7 

9 

62 

53 

8 

10 

Support  from  senior 
management 

9 

7.5 

54 

58 

7 

11 

Effective  management  of  risk 

10 

14.5 

46 

37 

6 

7 

Training  provision 

22 

16.5 

8 

32 

1 

6 

Note.  CSFs  are  ordered  for  project  managers  with  5  up  to  10  years  of  experience. 
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Similar  to  the  different  groups  of  experience,  the  project  managers  of  successful  projects 
and  critical  projects  have  a  very  close  percentage  of  respondents  selecting  the  same  CSFs  (Table 
16).  Although  these  project  managers  agreed  on  CSFs,  perhaps  the  successful  project  managers 
approached  these  CSFs  differently  than  the  unsuccessful  project  managers.  The  fact  CSFs  do  not 
seem  to  correlate  with  project  success  may  also  indicate  CSFs  have  no  relation  to  project 
success. 

One  distinct  difference  exists  in  building  an  effective  team,  which  55%  of  the  successful 
respondents  chose  versus  only  25%  of  the  critical  respondents.  The  successful  respondents,  on 
the  other  hand,  considered  support  from  senior  management  to  be  less  important  (36%),  while 
67%  of  the  critical  respondents  chose  it  as  a  CSF. 


Table  16 

CSF  Comparison  Between  Successful af7Qf  Critical  Projects 


CSFs 

Rank 

Success  Critical 

%of 

Respondents 
Success  Critical 

Raw  Data,  #  of 
Respondents 
Selecting  this 
CSF 

Success  Critical 

Adequate  funds/resources 

1 

3 

100 

88 

22 

21 

Clear  goals/objectives 

2 

3 

86 

88 

19 

21 

Realistic  schedule 

3 

1 

82 

92 

18 

22 

Clear  communication  channels 

Having  access  to  innovative/talented 

4 

5 

77 

75 

17 

18 

people 

5 

6 

73 

71 

16 

17 

End  user  commitment 

6 

3 

68 

88 

15 

21 

Effective  leadership/conflict  resolution 

7 

9 

59 

58 

13 

14 

Effective  management  of  risk 

8.5 

10 

55 

46 

12 

11 

Effective  team  building/motivation 

8.5 

15 

55 

25 

12 

6 

Support  from  stakeholder(s)/champion(s) 

11 

7.5 

41 

67 

9 

16 

Support  from  senior  management 

13.5 

7.5 

36 

67 

8 

16 

Note.  CSFs  are  ordered  for  successful  projects. 
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Limitations 


Stanton  and  Rogelberg  (2001)  reviewed  literature  on  the  pros  and  cons  of  online 
questionnaires.  Applicable  risks  to  such  research  include  access  control,  authentication,  multiple 
responding,  and  an  uncontrolled  response  environment.  The  online  questionnaire  for  this  study 
was  not  linked  to  any  other  website,  thereby  impeding  random  web  surfers  from  answering  the 
questionnaire.  To  maintain  anonymity,  the  respondents  did  not  have  to  authenticate  themselves. 
Authentication  was  not  a  concern  because  all  respondents  worked  in  software  project 
organizations,  regardless  of  whether  they  were  the  original  recipients  of  my  emails.  Therefore, 
their  answers  were  still  useful. 

Stanton  and  Rogelberg  (2001)  advise  how  to  minimize  the  possibility  of  someone 
responding  multiple  times:  1)  avoid  angering  the  respondents,  2)  request  one  response  per 
respondent,  and  3)  confirm  their  answers  before  submitting  to  the  database.  The  questionnaire 
and  invitation  were  designed  to  implement  their  recommendations.  The  uncontrolled  response 
environment  (the  respondent’s  situation  and  office  when  answering  the  items)  was  not  within  the 
study's  control,  and  Stanton  and  Rogelberg  concedes  that  “this  offers  no  worse  a  challenge  than 
standard  mail-return  survey  practices”  (p.  210). 

Stanton  and  Rogelberg  (2001)  also  note  the  danger  that  some  parts  of  the  respondent 
population  may  not  have  access  to  online  questionnaires,  which  would  bias  the  results  toward 
those  people  who  do  have  access. 

This  was  not  a  pressing  issue  for  this  study  because,  in  general,  all  people  working  on 
software  project  management  have  to  have  computer  access.  Furthermore,  Simsek  and  Veiga 
(2001)  note  “[Online  questionnaires]  can  prove  quite  beneficial  for  obtaining  opinions  related  to 
new  software.  Likewise,  because  most  large  firms  [like  the  Air  Force]  and  their 
managerial/professional  employees  have  access  to  e-mail,  sample  surveys  for  these  populations 
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are  possible”  (p.  97).  Simsek  and  Veiga,  therefore,  expect  software  project  managers  would 
likely  have  email  and  internet  access. 

The  section  of  the  questionnaire  eliciting  CSFs  had  a  limitation  because  it  may  not 
include  all  true  critical  success  factors  for  software  project  management..  To  minimize  this 
limitation,  respondents  had  the  option  to  add  in  CSFs  which  were  not  on  the  questionnaire  list 
but  that  the  respondents  thought  should  be  included. 

Archival  data  was  unavailable  to  cross-check  the  extent  to  which  the  respondents 
followed  through  on  how  they  prioritized  their  goals. 

The  data  analysis  could  be  more  refined  for  each  software  organization  if  the 
questionnaire  had  asked  respondents  to  designate  which  organization  they  worked  for.  Although 
Organization  A  and  B  used  similar  training  curriculum  and  terminology,  it  is  possible  they  assign 
different  responsibilities  to  their  software  project  managers. 

Bias  in  the  results  may  exist  because  some  invitation  recipients  did  not  respond  to  the 
questionnaire.  The  nonrespondents  may  have  had  different  perceptions  than  those  who  did 
respond. 

Summary 

The  respondents  provided  a  great  deal  of  data  to  use  in  answering  the  three  investigative 
questions.  Although  the  questions  can  be  answered  by  averaging  out  the  data,  an  analysis  within 
each  demographic  group  provided  an  even  more  refined  view  of  the  data  and  distinctions 
between  the  groups.  The  analysis  also  revealed  surprising  similarities,  such  as  between 
successful  and  critical  projects.  Limitations  in  the  research  acknowledge  this  study  is  imperfect. 
Nonetheless,  inferences  can  be  made  from  this  data  thanks  to  the  many  steps  taken  to  make  it 
valid  and  reliable. 
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V.  Conclusion 


Interpreting  the  Data  and  Drawing  Conclusions 

This  exploratory  study  began  with  the  following  three  investigative  questions: 

1.  How  do  Air  Force  software  project  managers  prioritize  the  three  traditional  measures  of 
project  success? 

2.  Does  high  user  interaction  in  Air  Force  software  projects  correlate  with  project  success? 

3.  Are  the  critical  success  factors  for  Air  Force  software  projects  different  from  private 
sector  projects? 

The  questionnaire  results  indicate  private  sector  research  on  these  constructs  is  mostly 
generalizable  to  the  two  surveyed  Air  Force  units,  specifically  Organization  Alpha  and 
Organization  Bravo.  Although  researchers  have  found  public  sector  differences  in  the 
procedures,  contracting,  and  documentation  of  software  acquisitions,  project  management  itself 
is  not  much  different  from  the  private  sector  for  these  two  organizations.  Therefore,  many  of  the 
private  sector  advances  in  project  management  should  be  applicable  to  these  Air  Force 
organizations,  as  well. 

With  the  first  investigative  question,  the  data  indicates  software  project  managers  in  these 
two  organizations  prefer  to  fulfill  requirements  over  budget  and  schedule  concerns.  This  is 
interesting  because  it  does  not  line  up  well  with  cost  as  an  independent  variable  (CAIV)  for 
major  automated  information  systems,  despite  directives  that  have  provided  guidance  for  CAIV 
since  1996.  Possibly,  software  project  managers  realize  they  can  always  deliver  a  project  on 
time  and  on  budget  without  fulfilling  requirements,  but  the  product  would  perform  poorly  and 
not  gain  customer  satisfaction.  Nonetheless,  cost  is  still  an  important  issue,  especially  because 
cost  overruns  can  provoke  the  cancellation  of  poorly  done  projects.  Therefore,  project  managers 
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should  reorient  the  traditional  success  criteria  toward  satisfying  all  three  with  trade-offs.  For 
example,  a  project  manager  and  customer  team  could  prioritize  40  requirements  for  a  software 
project.  As  money  and  time  run  out,  the  project  manager  could  disregard  the  lowest  priority 
requirements  in  favor  of  finishing  the  project  on  time,  on  budget,  and  fulfilling  those 
requirements  that  are  most  important  to  the  customer. 

With  the  second  investigative  question,  highly  successful  Air  Force  projects  had  strong 
user  interaction,  and  unsuccessful  projects  did  not.  Private  sector  research  has  recommended 
user  interaction  for  at  least  the  last  10  years.  However,  this  measure  is  apparently  not  a  rule  of 
thumb  in  Organization  Alpha  and  Organization  Bravo,  judging  from  the  24  critical  projects  and 
some  fair  projects  that  lacked  user  interaction.  But  the  strong  adjusted  r  square  of  0.735  and  low 
P  value  indicate  this  is  a  compelling  model  that  shows  the  importance  of  strong  user  interaction 
for  a  successful  project.  Every  large  project  with  a  separation  between  the  project  manager  and 
end-users  should  have  at  least  a  liaison  to  represent  the  end-users  and  the  customers  paying  for 
the  project. 

The  third  investigative  question  had  an  answer  that  suggests  a  partial  difference  between 
the  private  sector  and  the  two  Air  Force  organizations.  Although  the  Air  Force  software  project 
managers  and  private  sector  project  managers  shared  many  of  the  same  CSFs,  there  were  still 
some  distinct  differences.  These  differences  probably  come  from  the  different  organizational 
cultures  and  environments.  For  example,  Air  Force  project  managers  may  have  highly  valued 
“having  access  to  innovative/talented  people”  because  these  people  are  a  scarce  resource  in  the 
Air  Force.  Many  innovative/talented  people  periodically  move  to  another  assignment  to  progress 
in  their  careers  or  leave  the  military  to  work  elsewhere.  On  the  other  hand,  private  sector 
managers  have  the  freedom  to  hire  innovative/talented  people  for  competitive  salaries  and  then 
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just  as  easily  fire  them  when  the  economy  goes  into  a  recession.  So,  Air  Force  project  managers 
must  value  the  people  they  have,  while  private  sector  managers  may  consider  them  to  be  a  dime 
a  dozen. 

There  were  also  some  disparities  between  groupings  of  the  data.  While  most  groupings 
agreed  on  most  CSFs,  they  occasionally  had  distinct  differences.  The  successful  and 
unsuccessful  project  managers  were  most  remarkable  of  all,  though.  Both  strongly  agreed  on 
CSFs,  yet  the  CSFs  did  not  seem  to  have  any  predictive  value  in  achieving  project  success. 
Possibly  both  groups  of  software  project  managers  had  the  same  training  that  told  them  these 
CSFs  were  important,  but  only  the  successful  project  managers  were  good  at  using  these  CSFs. 
Or  it  could  be  that  the  CSF  construct  is  invalid  and  has  very  little  bearing  on  the  success  of  a 
project  for  these  two  organizations.  If  this  is  so,  then  CSFs  are  just  mythical  silver  bullets  that 
are  supposed  to  ensure  project  success,  and  the  real  answer  could  be  all  the  processes  involved 
with  producing  software,  as  per  the  various  capability  maturity  models. 

Research  Implications 

For  the  practitioner,  these  results  provide  information  to  make  better  decisions.  For 
example,  the  necessity  to  balance  the  trade-offs  of  not  reaching  all  goals  but  still  producing  a 
software  product  that  both  the  customer  and  user  will  accept  and  be  satisfied  with.  Good  user 
interaction  throughout  the  life  cycle  can  strengthen  user  satisfaction.  Consequently,  the  two  Air 
Force  organizations  may  desire  to  make  user  interaction  into  mandatory  policy.  And  this 
research  revealed  which  CSFs  are  valued  most  by  the  software  project  managers  of  Alpha  and 
Bravo.  With  these  CSFs  in  mind,  software  project  managers  for  these  organizations  can  allocate 
their  attention  appropriately  to  these  factors  as  they  progress  toward  a  successful  project. 
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For  academics,  this  research  explored  a  topic  that  has  many  precedents  in  the  private 
sector  and  no  precedents  in  the  public  sector,  specifically  software  development  and  maintenance 
within  Alpha  and  Bravo.  This  study  can  be  a  founding  stone  for  future  research  that  explores 
differences  in  the  CSFs.  Future  research  could  also  quantify  the  relationships  between  CSFs  and 
project  success  and  test  the  causative  nature  of  the  correlations  that  this  study  found.  It  could 
also  test  the  legitimacy  of  CSFs. 

Further  Research 

There  are  several  opportunities  for  further  research.  Different  research  methodologies 
and  questions  would  provide  more  insight  into  the  cost  overruns  and  other  failure  issues. 

While  this  questionnaire  focused  on  project  managers,  a  study  surveying  the  project  team 
members  could  further  test  generalizability.  White  and  Leifer  (1986)  surveyed  software  project 
team  members  in  the  private  sector,  whereas  other  researchers  focus  solely  on  project  managers. 
Their  results  indicate  project  team  members  consider  neither  user  participation  nor  senior 
management  support  (another  CSF)  to  be  vital  for  a  project's  success.  White  and  Leifer's  study 
on  project  team  members  is  one  of  few  focusing  on  project  team  members,  and  thus  it  invites 
further  study  to  support  their  surprising  results. 

A  case  study  on  the  new  Air  Force  personnel  information  system,  MILPDS  (formerly 
known  as  MilMod),  could  yield  useful  data  to  aid  the  Army’s  new  information  software  system. 
Literature  on  project  management  and  software  development  has  consistently  identified  failed 
information  systems  and  software  projects  as  a  common  problem  for  the  last  30  years.  Although 
corporations  and  the  government  have  made  many  information  systems  and  researchers  have 
studied  the  art  of  successful  software  project  management,  there  are  still  no  rules  to  guarantee 
project  success.  Public  sector  information  systems  pose  unique  difficulties  because  they  deal 
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with  many  more  employees  than  a  large  business  and  they  can  cost  much  more  money.  The 
military  must  learn  from  MilMod  and  the  new  Air  Force  civilian  system  to  ensure  the  Army's 
new  information  system  does  not  repeat  their  mistakes. 

The  comparisons  between  demographic  groups  may  also  lead  to  new  discoveries.  In 
particular,  the  lack  of  differences  between  the  successful  and  unsuccessful  projects  questions  the 
legitimacy  of  the  CSF  construct  for  Air  Force  software  project  management.  Or  perhaps 
moderators  are  influencing  the  impact  that  CSFs  are  supposed  to  make  in  promoting  project 
success. 
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Appendix  A,  Questionnaire  Invitation 


Dear  [Alpha/Bravo]  member, 

The  Air  Force  is  researching  software  project  success,  and  we  would  appreciate  your  help  with 
this. 

Software  project  success  has  been  a  notoriously  elusive  goal  for  the  private  sector.  Many 
projects  have  either  finished  over  time  and  over  budget  or  were  just  canceled.  Some  finished 
projects  still  do  not  meet  specifications  or  satisfy  the  customer.  For  these  reasons,  researchers 
have  published  many  studies  on  this  subject  in  the  private  sector. 

The  Air  Force  is  also  researching  project  success.  We  hope  to  learn  about  the  different  factors 
that  go  into  the  success  of  software  projects  and  how  project  managers  handle  some  of  these 
factors. 

Both  [Alpha]  and  [Bravo]  have  volunteered  to  survey  their  experts  on  this  topic.  You  are  an 
expert  on  software  project  management  because  of  your  insight  and  insider’s  knowledge  of  how 
it  is  done  in  the  Air  Force.  Even  members  new  to  software  project  management  have  a  unique 
understanding  and  background  that  contributes  to  this  study.  Although  some  of  our  projects  may 
not  be  completely  managed  and  developed  in  house,  [Alpha]  and  [Bravo]  members  still  see  how 
the  project  is  working  out. 

Therefore,  I  would  appreciate  your  help  by  answering  questions  on  software  project  success. 
Your  thoughts  would  shed  some  light  on  this  puzzling  topic.  All  answers  will  be  anonymous. 

For  your  convenience,  the  survey  is  online: 

http://en.afit.edu/env/spm/ 

Feel  free  to  email  me  at  [student@afit.edu]  if  you  would  like  to  hear  about  the  research  results  or 
if  you  have  any  questions.  I  am  a  graduate  student  at  the  Air  Force  Institute  of  Technology.  My 
advisor  is  [professor],  [professor@afit.edu]. 

Both  the  Air  Force  Personnel  Center  and  the  Union  have  approved  this  survey. 

Respectfully, 

[first  name,  last  name,  rank,  service] 

[office  symbol] 

[student@afit.edu] 
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Appendix  B,  Questionnaire 

Survey  on  Software  Project  Management 

Thank  you  for  taking  this  survey.  Your  candid  answers  will  help  research  on  successful  project 
management.  All  answers  will  be  anonymous. 

Credit  for  designing  some  of  this  survey  goes  to  Dr.  Jeffrey  Pinto  and  Dr.  D.K.  Slevin  (part  III) 
and  to  Dr.  Diana  White  and  Dr.  Joyce  Fortune  (part  IV). 


Part  I:  Background  and  Perspective 


1.  Rank: 

—select  one— 

•  E-l  to  E-4 

•  E-5  to  E-6 

•  E-7  to  E-9 

•  0-1  to  0-3 

•  0-4  to  0-6 

•  GS-ltoGS-5 

•  GS-6  to  GS-9 

•  GS-10  to  GS-14 

•  contractor 

•  other 

2.  How  many  years  have  you  worked  for  the  Air  Force? 

(less  than  1  year  =  0) 

3.  How  many  years  of  experience  do  you  have  with  software  engineering  and/or  project 
management?  (less  than  1  year  =  0) 

4.  What  role  do  you  fill  with  your  current  or  most  recent  software  project? 

—select  one — 

•  program  manager 

•  project  manager 

•  project  lead 

•  project  team  member 

•  consultant 

•  other  _ 

Any  comments  or  questions? _ 
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Part  II:  Project  Priorities 


With  each  question,  imagine  you  are  in  charge  of  a  particular  software  project,  which  could  deal 
with  personnel,  munitions,  accounting,  mapping,  or  communications.  The  three  traditional  goals 
of  projects  managers  are  to  finish  on  time,  on  budget,  and  meet  expectations.  But  sometimes,  not 
all  three  of  these  can  be  met. 

With  each  question,  prioritize  your  goals  by  selecting  your  top  priority.  Even  if  you  have  no 
experience  with  a  particular  project,  imagine  how  you  would  make  your  decisions.  For  the  ease 
of  answering  this  survey,  only  two  goals  need  to  be  considered  for  each  question.  You  will  see 
the  same  questions  and  responses  several  times.  Do  not  let  previous  responses  sway  your 
choices. 

All  of  these  projects  are  primarily  developed  by  the  government  or  contractors  (i.e.  no 
commercial  off  the  shelf  software).  For  the  purposes  of  this  survey,  designing,  developing,  and 
building  a  project  are  all  examples  of  creating  a  new  product  and  delivering  it  through  the 
lifecycle  to  implementation. 

Goals 

Meet  expectations:  finish  a  project  according  to  technical  specifications 
On  time:  finish  a  project  early  or  by  its  original  deadline  without  extensions 
On  budget:  finish  a  project  within  the  budget  without  any  requested  increases 


Project  Examples 

Personnel  system:  records  annual  performance  reviews 
Munitions  system:  maintains  inventory  of  munitions 

Accounting  system:  helps  automate  budgeting,  tax  preparation,  and  other  financial  work 
Base  map  system:  displays  maps  of  the  base,  terrain,  streets,  utilities,  and  traffic  patterns 
Wireless  communication  system:  securely  connects  laptops  with  base  network 

1 .  When  developing  a  wireless  communication  system,  which  is  more  important? 


on  budget  or 

2.  When  building  a  personnel  system,  which  is  more  important? 

meeting  expectations  or 

3.  When  designing  a  munitions  system,  which  is  more  important? 

on  time  or 

4.  When  developing  an  accounting  system,  which  is  more  important? 

on  budget  or 


meeting  expectations 


on  budget 


on  budget 


on  time 
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5.  When  building  a  wireless  communication  system,  which  is  more  important? 

on  time  or  on  budget 

6.  While  developing  a  personnel  system,  which  is  more  important? 

on  budget  or  on  time 

7.  While  designing  a  base  map  system,  which  is  more  important? 

meeting  expectations  or  on  budget 

8.  When  developing  an  accounting  system,  which  is  more  important? 

meeting  expectations  or  on  time 

9.  When  building  a  base  map  system,  which  is  more  important? 

on  time  or  on  budget 

10.  When  developing  a  munitions  system,  which  is  more  important? 

on  budget  or  meeting  expectations 

1 1 .  When  designing  a  wireless  communication  system,  which  is  more  important? 

on  time  or  meeting  expectations 

12.  When  developing  an  accounting  system,  which  is  more  important? 

meeting  expectations  or  on  budget 

13.  When  building  a  personnel  system,  which  is  more  important? 

meeting  expectations  or  on  time 

14.  While  developing  a  munitions  system,  which  is  more  important? 


on  time 


or  meeting  expectations 


15.  When  designing  a  base  map  system,  which  is  more  important? 

meeting  expectations  or  on  time 
Any  comments  or  questions? _ 
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Part  III:  User  and  Customer  Interaction 


With  this  third  part  of  the  survey,  consider  the  questions  with  your  current  or  most  recent 
software  project.  If  you  are  working  on  several  projects  simultaneously,  just  pick  one  of  your 
projects  for  these  questions. 

General  Project  Questions 

1 .  How  would  you  categorize  your  project? 

•  new  start  software  development 

•  sustaining  a  legacy  system 

•  other  _ 

2.  How  would  you  classify  it? 

•  commercial  off  the  shelf 

•  government  developed 

•  other  _ 

3.  Which  phase  is  your  project  at  in  the  software  lifecycle? 

•  requirements  gathering 

•  design 

•  implementation 

•  maintenance 

•  other 
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Customer  Project  Questions 


With  this  project  in  mind,  consider  the  following  statements.  Using  the  scale  below,  rate  each 
statement  according  to  the  degree  to  which  you  agree  with  the  statement  as  it  concerns  your 
project.  A  rating  of  5  indicates  that  the  statement  is  neutral  and  you  neither  agree  nor  disagree. 
A  rating  above  5  indicates  agreement  with  that  statement.  A  rating  below  5  indicates 
disagreement  with  the  statement. 

In  answering  the  following  questions,  these  definitions  apply: 

Customer:  the  agency  requesting  your  software 
Users:  the  people  who  will  use  your  software 


Strongly 

Disagree 

Neutral 

Strongly 

Agree 

0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 


The  customers  were  given  the  opportunity  to  provide  input  early  in  the  project  development 
stage. 

0123456789  10 

The  customers  are  kept  informed  of  the  project's  progress. 

0123456789  10 

The  value  of  the  project  has  been  discussed  with  the  customers. 

0123456789  10 

The  limitations  of  the  project  have  been  discussed  with  the  customers  (what  the  project  is  not 
designed  to  do). 

0123456789  10 

The  customers  were  told  whether  or  not  their  input  was  assimilated  into  the  project  plan. 

0123456789  10 

There  is  adequate  documentation  of  the  project  to  permit  easy  use  by  users  (instructions, 
etc.). 

0123456789  10 

Potential  users  have  been  contacted  about  the  usefulness  of  the  project. 

0123456789  10 

An  adequate  presentation  of  the  project  has  been  developed  for  customers. 

0123456789  10 
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Strongly 

Disagree 

Neutral 

Strongly 

Agree 

0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

9.  The  users  know  who  to  contact  when  problems  or  questions  arise. 

0123456789  10 

10.  Adequate  advanced  preparation  has  been  done  to  determine  how  best  to  “sell”  the  project  to 
users. 

0123456789  10 

1 1 .  The  results  (decisions  made,  information  received  and  needed,  etc)  of  planning  meetings  are 
published  and  distributed  to  applicable  personnel. 

0123456789  10 

12.  Individuals/groups  supplying  input  have  received  feedback  on  the  acceptance  or  rejection  of 
their  input. 

0123456789  10 

13.  When  the  budget  or  schedule  is  revised,  the  changes  and  the  reasons  for  the  changes  are 
communicated  to  customers. 

0123456789  10 

14.  The  reasons  for  any  changes  to  existing  policies/procedures  have  been  explained  to 
customers. 

0123456789  10 

15.  All  groups  affected  by  the  project  know  how  to  make  problems  known  to  me. 

0123456789  10 

16.  This  project  has/will  come  in  on  schedule. 

0123456789  10 

17.  This  project  has/will  come  in  on  budget. 

0  1  2  3  4  5  6  7  8  9  10 

18.  The  project  that  has  been  developed  works  (or,  if  still  being  developed,  looks  as  if  it  will 

work).  0123456789  10 
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Strongly 

Disagree 

Neutral 

Strongly 

Agree 

0 

1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

19.  The  project  will  be/is  used  by  its  intended  users. 

0123456789  10 

20.  This  project  has/will  directly  benefit  the  intended  users:  either  through  increasing  efficiency 
or  employee  effectiveness. 

0123456789  10 

21.  Given  the  problem  for  which  it  was  developed,  this  project  seems  to  do  the  best  job  of 
solving  that  problem —  i.e.,  it  was  the  best  choice  among  the  set  of  alternatives. 

0123456789  10 

22.  Important  customers,  directly  affected  by  this  project,  will  make  use  of  it. 

0123456789  10 

23. 1  am/was  satisfied  with  the  process  by  which  this  project  is  being/was  completed. 

0123456789  10 

24.  We  are  confident  that  non-technical  start-up  problems  will  be  minimal,  because  the  project 
will  be  readily  accepted  by  its  intended  users. 

0123456789  10 

25.  Use  of  this  project  has/will  directly  lead  to  improved  or  more  effective  decision  making  or 
performance  for  the  users. 

0123456789  10 

26.  This  project  will  have  a  positive  impact  on  those  who  make  use  of  it. 

0123456789  10 

27.  The  results  of  this  project  represent  a  definite  improvement  in  performance  over  the  way  end- 
users  used  to  perform  these  activities. 

0123456789  10 


Any  comments  or  questions? 
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Part  IV:  Critical  Success  Factors 


Which  10  of  these  23  factors  do  you  consider  most  critical  to  the  success  of  your  project?  Which 
factors  are  essential  to  success?  Please  mark  your  top  10;  no  rank  ordering  is  necessary.  In  the 
textbox  below,  feel  free  to  note  any  other  factors  that  should  be  added  to  this  list  as  part  of  your 
list  of  10. 


_  Adequate  funds/resources 

_  Appreciating  the  effect  of  human  error 

_  Clear  communication  channels 

_  Clear  goals/objectives 

_  Considering  multiple  views  of  project 

_  Contextual  awareness 

_  Effective  leadership/conflict  resolution 

_  Effective  management  of  risk 

_  Effective  monitoring  and  feedback 

_  Effective  team  building/motivation 

_  End  user/customer  commitment 

_  Flexible  approach  to  change 

_  Having  a  clear  project  boundary 

_  Having  access  to  innovative/talented  people 

_  Having  relevant  past  experience 

_  Provision  of  planning  and  control  systems 

_  Realistic  schedule 

_  Recognizing  complexity 

_  Support  from  senior  management 

_  Support  from  stakeholder(s)/champion(s) 

_  Taking  account  of  external  influences 

_  Taking  account  of  past  experience 

_  Training  provision 

Other  factor(s): _ 


Any  comments  or  questions? 


Once  again,  thank  you  for  your  help  with  this  study.  If  you  would  like  an  executive  summary  of 
my  research  results,  please  email  me  at  [student@afit.edu] 
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Appendix  C,  Demographics 

Raw  Data  on  Respondents 

Respondents  #2,  6,  16,  21,  24,  and  69  only  completed  Parts  I  and  II  of  the  questionnaire. 
Respondents  received  ID  numbers  on  first  come  basis.  #1  was  the  first  person  to  respond. 


Years  Years  of  Experience  in  Project 
Working  Management  and/or  Software 

ID  # _ Rank _ For  AF _ Engineering _ Role 


1 

GS-IOto  GS-14 

23 

14 

project  lead 

2 

GS-IOto  GS-14 

12 

5 

other 

3 

GS-IOto  GS-14 

17 

7 

project  lead 

4 

GS-IOto  GS-14 

28 

3 

project  team  member 

5 

GS-IOto  GS-14 

16 

12 

program  manager 

6 

GS-IOto  GS-14 

34 

27 

project  manager 

7 

GS-IOto  GS-14 

25 

20 

project  team  member 

8 

GS-IOto  GS-14 

8 

3 

program  manager 

9 

GS-IOto  GS-14 

25 

25 

project  lead 

10 

GS-IOto  GS-14 

14 

10 

project  manager 

11 

GS-IOto  GS-14 

23 

12 

other 

12 

GS-IOto  GS-14 

36 

26 

project  manager 

13 

GS-IOto  GS-14 

16 

5 

project  lead 

14 

GS-IOto  GS-14 

24 

5 

project  team  member 

15 

GS-IOto  GS-14 

29 

8 

project  manager 

16 

GS-IOto  GS-14 

36 

8 

project  lead 

17 

GS-IOto  GS-14 

36 

14 

project  manager 

18 

GS-IOto  GS-14 

30 

25 

project  lead 

19 

GS-IOto  GS-14 

22 

22 

project  lead 

20 

GS-IOto  GS-14 

20 

17 

program  manager 

21 

GS-IOto  GS-14 

24 

19 

program  manager 

22 

GS-IOto  GS-14 

21 

11 

project  manager 

23 

0-1  to  0-3 

19.5 

2 

program  manager 

24 

GS-IOto  GS-14 

32 

10 

program  manager 

25 

GS-IOto  GS-14 

27 

10 

program  manager 

26 

other 

28 

22 

program  manager 

27 

GS-IOto  GS-14 

40 

17 

project  manager 

28 

GS-IOto  GS-14 

29 

3 

program  manager 

29 

GS-IOto  GS-14 

24 

20 

program  manager 

30 

GS-IOto  GS-14 

35 

5 

other 

31 

GS-IOto  GS-14 

8.5 

4 

project  manager 
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ID# 

Rank 

Years 
Working 
For  AF 

Years  of  Experience  in 
Project  Management  and/or 
Software  Engineering 

Role 

32 

E-7  to  E-9 

20 

6 

project  manager 

33 

GS-IOto  GS-14 

30 

4 

project  manager 

34 

0-1  to  0-3 

0 

0 

project  manager 

35 

GS-IOto  GS-14 

26 

16 

program  manager 

36 

0-1  to  0-3 

11 

2 

program  manager 

37 

GS-IOto  GS-14 

29 

10 

program  manager 

38 

GS-IOto  GS-14 

2 

10 

program  manager 

39 

GS-IOto  GS-14 

29 

10 

program  manager 

40 

GS-IOto  GS-14 

15 

14 

program  manager 

41 

E-7  to  E-9 

22 

6 

project  manager 

42 

GS-IOto  GS-14 

25 

8 

program  manager 

43 

GS-IOto  GS-14 

30 

20 

project  manager 

44 

0-1  to  0-3 

15 

2 

program  manager 

45 

0-1  to  0-3 

9 

0 

project  manager 

46 

GS-IOto  GS-14 

20 

18 

project  manager 

47 

0-1  to  0-3 

5 

0 

program  manager 

48 

GS-IOto  GS-14 

31 

20 

project  manager 

49 

0-1  to  0-3 

3 

1 

project  manager 

50 

GS-IOto  GS-14 

23 

10 

program  manager 

51 

0-1  to  0-3 

10.5 

4 

program  manager 

52 

GS-IOto  GS-14 

39 

20 

project  manager 

53 

GS-IOto  GS-14 

27 

5 

other 

54 

GS-IOto  GS-14 

26 

6 

program  manager 

55 

GS-IOto  GS-14 

26 

20 

program  manager 

56 

0-1  to  0-3 

4 

2 

program  manager 

57 

0-4  to  0-6 

6 

3 

project  manager 

58 

GS-IOto  GS-14 

42 

5 

other 

59 

GS-IOto  GS-14 

19 

15 

program  manager 

60 

GS-IOto  GS-14 

17 

8 

project  manager 

61 

0-1  to  0-3 

3 

1 

project  manager 

62 

GS-IOto  GS-14 

23 

15 

program  manager 

63 

0-1  to  0-3 

5 

2 

project  manager 

64 

GS-IOto  GS-14 

25 

3 

project  manager 

65 

GS-IOto  GS-14 

23 

15 

project  manager 

66 

GS-IOto  GS-14 

22 

22 

project  lead 

67 

GS-IOto  GS-14 

15 

5 

program  manager 

68 

GS-IOto  GS-14 

14 

14 

program  manager 

69 

GS-IOto  GS-14 

37 

0 

other 

70 

GS-IOto  GS-14 

17 

1 

project  lead 

Average: 

21.5 

10.2 
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Appendix  D  (continued) 


ID# 

Personnel 

T  B  ME 

Munitions 

T  B  ME 

Accounting 

T  B  ME 

Base  Map 

T  B  ME 

T 

Wireless 

B  ME 

36 

1 

2 

1 

2 

2 

1 

1 

2 

1 

2 

37 

1 

2 

1 

2 

2 

1 

1 

2 

1 

2 

38 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

39 

1 

1 

1 

1 

2 

1 

2 

1 

2 

1 

2 

40 

1 

2 

1 

2 

1 

2 

2 

1 

1 

2 

41 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

42 

2 

1 

1 

2 

1 

2 

1 

2 

1 

2 

43 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

44 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

45 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

46 

1 

2 

1 

2 

1 

2 

2 

1 

1 

2 

47 

1 

2 

1 

2 

2 

1 

2 

1 

1 

1 

1 

48 

1 

1 

1 

1 

2 

1 

2 

2 

1 

1 

2 

49 

1 

1 

1 

1 

2 

2 

1 

1 

2 

1 

2 

50 

2 

1 

1 

2 

1 

1 

1 

1 

2 

1 

2 

51 

2 

1 

1 

1 

1 

2 

1 

1 

2 

2 

1 

52 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

53 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

54 

1 

2 

1 

2 

2 

1 

1 

2 

1 

2 

55 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

56 

1 

2 

2 

1 

2 

1 

1 

2 

1 

1 

1 

57 

2 

1 

1 

2 

2 

1 

1 

2 

1 

2 

58 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

59 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

60 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

61 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

62 

1 

1 

1 

1 

2 

1 

2 

1 

1 

1 

1 

1 

1 

63 

1 

2 

1 

2 

2 

1 

1 

2 

1 

2 

64 

2 

1 

2 

1 

2 

1 

1 

2 

1 

2 

65 

2 

1 

1 

2 

1 

2 

2 

1 

2 

1 

66 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

67 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

68 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

69 

1 

2 

2 

1 

1 

2 

1 

2 

1 

2 

70 

1 

2 

1 

2 

1 

2 

1 

2 

1 

2 

Totals 

49 

43 

118 

62 

22 

126 

47 

46 

117 

43 

56 

111 

53 

35 

122 

61 


Appendix  E,  Project  Description 


ID 

£_ 

1 

3 

4 

5 

7 

8 
9 

10 

11 

12 

13 

14 

15 

17 

18 

19 

20 
22 
23 

25 

26 

27 

28 

29 

30 

31 

32 

33 

34 

35 


New  Software  or  Legacy _ 

sustaining  a  legacy  system 
new  start  software  development 
new  start  software  development 
new  start  software  development 
sustaining  a  legacy  system 
other:  sustainment  &  modernization 
new  start  software  development 
sustaining  a  legacy  system 
sustaining  a  legacy  system 
sustaining  a  legacy  system 
other:  modernizing  legacy 
sustaining  a  legacy  system 
new  start  software  development 
other 

other:  modernization 
sustaining  a  legacy  system 
new  start  software  development 
sustaining  a  legacy  system 
sustaining  a  legacy  system 
sustaining  a  legacy  system 
other:  new  start  &  modernization 
sustaining  a  legacy  system 
sustaining  a  legacy  system 
sustaining  a  legacy  system 
sustaining  a  legacy  system 
sustaining  a  legacy  system 
sustaining  a  legacy  system 
new  start  software  development 
new  start  software  development 
sustaining  a  legacy  system 


Developer _ 

other:  contractor  developed 
government  developed 
commercial  off  the  shelf 
other:  contractor  developed 
other:  contractor  developed 
government  developed 
other:  contractor  developed 
other:  both 

other:  contractor  developed 
government  developed 
other:  contractor  developed 
government  developed 
government  developed 
government  developed 
other:  contractor  developed 
government  developed 
commercial  off  the  shelf 
government  developed 
government  developed 
government  developed 
other:  COTS/GOTS/developed 
government  developed 
government  developed 
government  developed 
government  developed 
government  developed 
government  developed 
government  developed 
government  developed 
government  developed 


Lifecycle  Phase _ 

maintenance 

implementation 

implementation 

other:  development 

maintenance 

maintenance 

implementation 

maintenance 

maintenance 

maintenance 

requirements  gathering 

maintenance 

implementation 

maintenance 

maintenance 

maintenance 

requirements  gathering 

implementation 

maintenance 

maintenance 

other:  all  of  the  above 

maintenance 

maintenance 

maintenance 

maintenance 

other:  all  of  the  above 

maintenance 

implementation 

implementation 

maintenance 
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Appendix  E  (continued). 


ID 

# 

36 

37 

38 

39 

40 

41 

42 

43 

44 

45 

46 

47 

48 

49 

50 

51 

52 

53 

54 

55 

56 

57 

58 

59 

60 
61 
62 

63 

64 

65 

66 

67 

68 
70 


New  Software  or  Legacy _ 

other:  both 
other:  modernization 
sustaining  a  legacy  system 
sustaining  a  legacy  system 
sustaining  a  legacy  system 
sustaining  a  legacy  system 
sustaining  a  legacy  system 
sustaining  a  legacy  system 
other:  sustainment  of  a  fairly  new  system 
new  start  software  development 
other:  modifying  an  existing  system 
other:  helpdesk  management 
sustaining  a  legacy  system 
new  start  software  development 
sustaining  a  legacy  system 
new  start  software  development 
sustaining  a  legacy  system 
sustaining  a  legacy  system 
sustaining  a  legacy  system 
sustaining  a  legacy  system 
new  start  software  development 
new  start  software  development 
sustaining  a  legacy  system 
new  start  software  development 
new  start  software  development 
sustaining  a  legacy  system 
sustaining  a  legacy  system 
other:  transitioning  to  sustainment 
sustaining  a  legacy  system 
sustaining  a  legacy  system 
sustaining  a  legacy  system 
new  start  software  development 
new  start  software  development 
other:  eBiz  initiative 


Developer _ 

government  developed 
government  developed 
government  developed 
government  developed 
government  developed 
government  developed 
other 

government  developed 
government  developed 
commercial  off  the  shelf 
other:  contractor  developed 
other:  both 

government  developed 
commercial  off  the  shelf 
government  developed 
government  developed 
government  developed 
government  developed 
government  developed 
government  developed 
government  developed 
commercial  off  the  shelf 
government  developed 
other:  contractor  developed 
other:  both 

government  developed 
government  developed 
other:  both 

government  developed 
government  developed 
government  developed 
commercial  off  the  shelf 
commercial  off  the  shelf 
other:  both 


Lifecycle  Phase _ 

design 

design 

maintenance 

requirements  gathering 

maintenance 

maintenance 

maintenance 

implementation 

maintenance 

design 

maintenance 

other:  all 

maintenance 

maintenance 

maintenance 

design 

maintenance 

maintenance 

maintenance 

maintenance 

implementation 

design 

maintenance 

requirements  gathering 

requirements  gathering 

implementation 

maintenance 

other:  sustain/  req's 

maintenance 

maintenance 

maintenance 

design 

requirements  gathering 
implementation 
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Appendix  F,  Project  Success  Scores 


ID# 

User  Interaction  CSF  Items 

Project  Success  Items 

Total 

Score 

Client  Consultation 

12  3  4 

5 

Client  Acceptance 
6  7  8  9 

10 

11 

Communication 

12  13  14 

15 

16 

Project  Score 
17  18  19 

20 

21 

22 

Client  Score 

23  24  25 

26 

27 

1 

9 

9 

7 

6 

9 

9 

5 

9 

9 

5 

7 

5 

8 

8 

9 

5 

5 

9 

9 

9 

9 

9 

9 

9 

9 

9 

7 

98 

3 

8 

10 

10 

8 

8 

3 

8 

6 

8 

6 

8 

7 

7 

6 

6 

2 

2 

9 

10 

10 

5 

7 

7 

4 

9 

9 

9 

83 

4 

10 

10 

10 

10 

8 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

8 

9 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

117 

5 

10 

10 

10 

10 

10 

9 

10 

8 

10 

8 

10 

9 

10 

9 

10 

8 

9 

10 

10 

10 

8 

10 

10 

10 

10 

10 

10 

115 

7 

8 

8 

8 

6 

7 

8 

8 

8 

9 

8 

9 

8 

10 

9 

9 

5 

8 

8 

10 

10 

10 

10 

10 

9 

9 

9 

9 

107 

8 

10 

10 

10 

10 

10 

6 

7 

8 

9 

7 

10 

9 

10 

10 

6 

6 

6 

9 

10 

9 

9 

9 

8 

9 

9 

9 

10 

103 

9 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

5 

9 

9 

9 

10 

10 

10 

10 

10 

9 

10 

10 

10 

10 

117 

10 

8 

9 

8 

7 

7 

8 

8 

8 

9 

7 

8 

7 

9 

8 

8 

4 

4 

8 

9 

9 

8 

9 

7 

6 

8 

8 

8 

88 

11 

7 

8 

7 

7 

8 

6 

6 

8 

10 

7 

7 

6 

4 

4 

4 

6 

4 

8 

8 

8 

9 

8 

6 

6 

6 

7 

8 

84 

12 

10 

9 

9 

9 

8 

10 

10 

10 

8 

10 

7 

7 

9 

8 

9 

8 

8 

9 

9 

9 

9 

9 

9 

9 

9 

8 

8 

104 

13 

5 

5 

8 

4 

4 

4 

7 

5 

7 

5 

5 

8 

7 

7 

6 

5 

5 

5 

7 

7 

5 

6 

6 

6 

7 

7 

7 

73 

14 

2 

6 

3 

9 

5 

8 

8 

9 

10 

9 

7 

4 

8 

9 

9 

10 

10 

10 

10 

8 

9 

9 

10 

9 

8 

7 

8 

108 

15 

2 

5 

5 

5 

8 

9 

7 

9 

10 

7 

5 

8 

8 

8 

9 

2 

5 

10 

10 

10 

10 

10 

10 

3 

10 

10 

10 

100 

17 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

120 

18 

9 

9 

9 

9 

9 

8 

9 

8 

8 

8 

9 

9 

9 

2 

8 

9 

8 

9 

9 

9 

9 

9 

8 

9 

9 

9 

9 

106 

19 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

120 

20 

10 

10 

8 

10 

10 

9 

9 

10 

10 

7 

10 

8 

10 

10 

10 

8 

8 

10 

10 

10 

10 

10 

10 

9 

10 

10 

10 

115 

22 

10 

10 

10 

10 

10 

8 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

8 

8 

8 

8 

10 

8 

10 

10 

10 

110 

23 

9 

8 

10 

9 

10 

10 

1 

10 

10 

5 

10 

10 

10 

5 

9 

8 

3 

10 

10 

10 

5 

9 

10 

8 

9 

10 

10 

102 

25 

10 

8 

10 

10 

8 

3 

10 

9 

10 

3 

5 

5 

6 

6 

10 

7 

7 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

114 

26 

10 

10 

10 

7 

10 

8 

10 

10 

10 

7 

10 

8 

10 

8 

10 

6 

6 

9 

10 

10 

10 

10 

6 

6 

10 

10 

10 

103 

27 

8 

10 

10 

8 

8 

9 

7 

6 

10 

7 

10 

10 

10 

10 

10 

9 

9 

10 

10 

10 

8 

9 

9 

10 

9 

9 

10 

112 

28 

10 

10 

10 

10 

10 

7 

10 

9 

10 

8 

7 

10 

10 

10 

10 

8 

2 

7 

10 

6 

8 

9 

7 

8 

8 

8 

8 

89 

29 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

120 

30 

7 

7 

7 

5 

5 

4 

5 

4 

9 

7 

8 

6 

7 

7 

8 

7 

7 

7 

7 

7 

7 

7 

7 

7 

7 

7 

7 

84 

31 

10 

10 

10 

10 

10 

8 

7 

10 

10 

5 

10 

10 

10 

10 

8 

8 

8 

9 

10 

10 

10 

10 

4 

9 

9 

9 

7 

103 

32 

5 

8 

8 

8 

8 

7 

5 

8 

9 

5 

9 

9 

9 

9 

9 

7 

7 

9 

7 

8 

8 

8 

6 

5 

8 

8 

9 

90 

33 

10 

8 

8 

8 

8 

8 

8 

8 

8 

8 

8 

8 

8 

8 

8 

5 

8 

8 

8 

8 

8 

8 

8 

8 

8 

8 

8 

93 

34 

6 

10 

10 

7 

6 

8 

10 

8 

9 

7 

9 

8 

9 

9 

8 

10 

10 

10 

10 

9 

4 

6 

9 

7 

8 

6 

7 

96 

35 

10 

10 

10 

10 

10 

10 

7 

7 

10 

7 

7 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

120 

36 

10 

10 

10 

8 

8 

8 

8 

7 

10 

9 

8 

8 

8 

8 

8 

9 

5 

9 

8 

10 

9 

9 

9 

9 

9 

9 

10 

105 

Appendix  F  (continued). 


ID# 

User  Interaction  CSF  Items 

Project  Success  Items 

Total  Score 

Client  Consultation 

1  2  3  4  5 

Client  Acceptance 

6  7  8  9  10 

11 

Communication 

12  13  14 

15 

16 

Project  Score 
17  18  19 

20 

21 

22 

Client  Score 

23  24  25 

26 

27 

37 

10 

10 

10 

10 

10 

10 

10 

10 

10 

8 

10 

10 

10 

10 

10 

7 

8 

8 

10 

9 

9 

8 

8 

9 

9 

9 

9 

103 

38 

8 

8 

10 

8 

9 

7 

9 

8 

9 

7 

6 

6 

8 

8 

8 

6 

7 

7 

8 

8 

8 

8 

6 

7 

7 

8 

8 

88 

39 

10 

10 

10 

9 

9 

9 

9 

9 

10 

8 

9 

9 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

9 

9 

10 

10 

10 

118 

40 

10 

10 

10 

10 

10 

8 

10 

10 

10 

10 

10 

10 

10 

10 

10 

3 

8 

10 

10 

10 

9 

10 

8 

9 

9 

9 

10 

105 

41 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

120 

42 

8 

8 

7 

6 

7 

6 

7 

7 

8 

7 

7 

7 

7 

6 

6 

6 

6 

7 

8 

7 

7 

7 

7 

7 

7 

8 

7 

84 

43 

9 

8 

8 

8 

8 

8 

8 

6 

10 

8 

8 

8 

8 

8 

10 

2 

5 

10 

10 

10 

10 

9 

2 

8 

9 

10 

9 

94 

44 

10 

10 

10 

10 

10 

7 

10 

9 

10 

8 

9 

9 

10 

10 

8 

9 

9 

10 

10 

10 

10 

10 

10 

8 

10 

10 

10 

116 

45 

2 

2 

5 

2 

3 

1 

2 

4 

6 

4 

3 

4 

3 

5 

6 

5 

6 

5 

5 

5 

3 

5 

3 

5 

5 

5 

5 

57 

46 

10 

9 

9 

9 

9 

10 

10 

9 

10 

10 

10 

10 

8 

9 

9 

8 

9 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

117 

47 

10 

9 

10 

9 

9 

9 

8 

9 

10 

9 

9 

9 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

120 

48 

4 

9 

9 

7 

5 

3 

5 

9 

10 

9 

9 

5 

3 

3 

10 

10 

10 

9 

9 

7 

9 

9 

10 

8 

8 

8 

8 

105 

49 

7 

8 

8 

5 

6 

6 

7 

7 

10 

9 

10 

8 

8 

7 

10 

7 

6 

7 

7 

8 

7 

7 

7 

8 

8 

8 

7 

87 

50 

8 

9 

10 

10 

9 

9 

10 

10 

10 

9 

9 

9 

8 

9 

10 

3 

6 

9 

9 

10 

9 

10 

9 

9 

10 

10 

10 

104 

51 

10 

10 

10 

10 

10 

9 

6 

10 

10 

2 

9 

7 

10 

10 

9 

10 

10 

10 

8 

10 

9 

9 

6 

8 

10 

10 

10 

110 

52 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

8 

8 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

116 

53 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

120 

54 

9 

9 

9 

5 

6 

5 

6 

5 

7 

7 

8 

5 

5 

5 

7 

10 

10 

5 

5 

5 

8 

8 

8 

7 

7 

7 

7 

87 

55 

6 

5 

6 

5 

7 

7 

6 

8 

9 

8 

6 

8 

9 

9 

7 

9 

9 

9 

9 

8 

7 

7 

6 

6 

7 

7 

7 

91 

56 

7 

5 

6 

6 

6 

8 

6 

7 

6 

6 

6 

7 

7 

6 

5 

6 

2 

2 

6 

6 

5 

6 

4 

4 

5 

4 

6 

56 

57 

10 

8 

10 

10 

10 

10 

10 

10 

10 

9 

7 

5 

5 

5 

5 

9 

0 

7 

10 

10 

10 

10 

8 

5 

10 

10 

10 

99 

58 

9 

9 

5 

7 

9 

5 

5 

5 

9 

4 

6 

8 

5 

5 

9 

6 

6 

8 

8 

7 

9 

9 

1 

9 

9 

9 

9 

90 

59 

10 

9 

9 

7 

10 

9 

5 

9 

10 

7 

9 

9 

9 

9 

9 

9 

8 

8 

8 

8 

10 

10 

10 

8 

8 

8 

8 

103 

60 

8 

10 

10 

9 

10 

5 

7 

6 

7 

4 

6 

8 

8 

8 

8 

7 

7 

7 

8 

8 

9 

7 

7 

8 

8 

8 

5 

89 

61 

4 

4 

4 

4 

4 

10 

4 

4 

5 

1 

1 

2 

0 

4 

8 

0 

0 

10 

10 

10 

9 

9 

0 

5 

10 

10 

10 

83 

62 

10 

10 

10 

10 

9 

9 

3 

9 

9 

3 

9 

8 

10 

10 

9 

9 

9 

9 

9 

10 

10 

10 

9 

9 

9 

9 

10 

112 

63 

10 

10 

7 

7 

10 

6 

5 

7 

9 

0 

3 

1 

7 

7 

8 

9 

9 

9 

4 

3 

3 

2 

9 

4 

3 

6 

7 

68 

64 

10 

10 

10 

10 

10 

10 

8 

8 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

120 

65 

7 

7 

7 

7 

7 

7 

7 

7 

9 

8 

7 

7 

7 

7 

8 

7 

7 

7 

7 

7 

7 

7 

7 

7 

7 

7 

7 

84 

66 

10 

10 

10 

10 

10 

10 

10 

10 

10 

5 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

10 

120 

67 

10 

9 

9 

9 

8 

9 

9 

9 

9 

9 

9 

9 

9 

9 

7 

8 

9 

10 

9 

9 

8 

8 

7 

5 

7 

8 

8 

96 

68 

5 

5 

6 

5 

5 

5 

10 

6 

7 

6 

5 

9 

5 

5 

5 

5 

5 

5 

10 

5 

5 

5 

5 

5 

5 

5 

5 

65 

70 

10 

10 

10 

9 

9 

10 

9 

5 

9 

9 

5 

9 

9 

9 

5 

7 

7 

7 

10 

9 

10 

10 

10 

10 

10 

9 

10 

109 

Appendix  G,  Pinto  and  Sieving  Project  Scores 


Status 

Percentile 

Score 

Raw  Score 

Project 

Client 

Total 

%  of 

Individuals 
Scoring  Lower 

Consultation 

Acceptance 

Communication 

Good 

100 

49 

50 

50 

50 

70 

120 

90 

46 

45 

45 

49 

66 

115 

80 

42 

43 

41 

46 

63 

109 

Fair 

70 

40 

41 

38 

44 

60 

104 

60 

38 

39 

36 

43 

59 

102 

50 

35 

37 

35 

41 

56 

97 

40 

32 

35 

33 

40 

54 

94 

Critical 

30 

30 

33 

32 

37 

50 

87 

/  Failed 

20 

27 

30 

28 

34 

47 

81 

10 

20 

26 

25 

31 

44 

75 

0 

9 

7 

6 

7 

19 

26 

Note.  Combination  of  information  from  Slevin  and  Pinto  (1986)  and  Pinto  and  Slevin  (1988).  Pinto  and  Slevin 
reduced  questionnaire  from  10  items  to  5  items  per  construct.  Parts  of  questionnaire  had  7-point  Likert  scales 
(1 988)  and  other  parts  had  1 1  -point  Likert  scales  (1986).  Converted  all  items  to  1 1  -point  Likert  scale. 
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Appendix  H,  Mean  Responses  and  Range  to  User  Interaction  and  Project  Success 


Subject 

Item 

Range 

Min 

Max 

Mean 

Std.  Dev 

Variance 

N 

User  Consultation 

1 

8 

2 

10 

8.5 

2.2 

4.9 

64 

2 

8 

2 

10 

8.7 

1.8 

3.3 

64 

3 

7 

3 

10 

8.7 

1.8 

3.2 

64 

4 

8 

2 

10 

8.2 

2.0 

4.0 

64 

5 

7 

3 

10 

8.4 

1.9 

3.5 

64 

User  Acceptance 

6 

9 

1 

10 

7.8 

2.2 

4.8 

64 

7 

9 

1 

10 

7.8 

2.2 

5.0 

64 

8 

6 

4 

10 

8.2 

1.8 

3.2 

64 

9 

5 

5 

10 

9.2 

1.2 

1.4 

64 

10 

10 

0 

10 

7.3 

2.4 

5.6 

64 

Communication 

11 

9 

1 

10 

8.1 

2.1 

4.3 

64 

12 

9 

1 

10 

8.0 

2.1 

4.3 

64 

13 

10 

0 

10 

8.3 

2.1 

4.6 

64 

14 

8 

2 

10 

8.1 

2.1 

4.4 

64 

15 

6 

4 

10 

8.5 

1.6 

2.7 

64 

Project  Success 

16 

10 

0 

10 

7.4 

2.5 

6.0 

64 

17 

10 

0 

10 

7.3 

2.6 

6.8 

64 

18 

8 

2 

10 

8.7 

1.7 

2.8 

64 

19 

6 

4 

10 

9.0 

1.4 

2.1 

64 

20 

7 

3 

10 

8.8 

1.6 

2.6 

64 

User  Interaction 

21 

7 

3 

10 

8.5 

1.9 

3.5 

64 

Success 

22 

8 

2 

10 

8.7 

1.6 

2.6 

64 

23 

10 

0 

10 

8.0 

2.4 

5.7 

64 

24 

7 

3 

10 

8.0 

1.9 

3.7 

64 

25 

7 

3 

10 

8.7 

1.6 

2.4 

64 

26 

6 

4 

10 

8.7 

1.5 

2.2 

64 

27 

5 

5 

10 

8.8 

1.5 

2.2 

64 

Cumulative  (raw  data) 

Consulting 

1  to  5 

36 

14 

50 

42.5 

8.7 

75.6 

64 

Acceptance 

5  to  10 
10  to 

33 

17 

50 

40.4 

7.5 

55.8 

64 

Communication 

15 

35 

15 

50 

41.0 

8.3 

68.6 

64 

15  to 

Procect  Score 

20 

28 

22 

50 

41.2 

6.9 

48.2 

64 

20  to 

Client  Score 

27 

39 

31 

70 

59.3 

10.1 

102.4 

64 

Total  Score 

64 

56 

120 

100.5 

16.2 

261.0 

64 

Note.  Pinto  and  Slevin's  (1988)  items  on  Project  Success  asked  about  achievement  of  the  traditional 
success  criteria  of  finishing  on  budget,  on  time,  and  fulfilling  all  requirements. 
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Appendix  I,  CSF  Selection,  Overall 


Raw  Data  from  this  Study  and  White  &  Fortune,  2002 


CSFs 

White  &  Fortune,  2002 

Research  Results* 

Adequate  funds/resources 

164 

60 

Appreciating  the  effect  of  human  error 

53 

4 

Clear  communication  channels 

144 

46 

Clear  goals/objectives 

206 

56 

Considering  multiple  views  of  project 

47 

6 

Contextual  awareness 

94 

5 

Effective  leadership/conflict  resolution 

138 

32 

Effective  management  of  risk 

117 

34 

Effective  monitoring  and  feedback 

135 

19 

Effective  team  building/motivation 

117 

22 

End  user  commitment 

159 

51 

Flexible  approach  to  change 

133 

22 

Having  a  clear  project  boundary 

2 

26 

Having  access  to  innovative/talented  people 

8 

41 

Having  relevant  past  experience 

3 

22 

Provision  of  planning  and  control  systems 

88 

6 

Realistic  schedule 

185 

55 

Recognizing  complexity 

121 

22 

Support  from  senior  management 

176 

35 

Support  from  stakeholder(s)/champion(s) 

3 

34 

Taking  account  of  external  influences 

120 

10 

Taking  account  of  past  experience 

121 

12 

Training  provision 

98 

16 

*Four  respondents  added  1  CSF  each  to  the  list  as  part  of  their  total  selection  of  10  CSFs. 
Verbatim _ 

End  user/customer  involvement/participation,  and  clearly  understanding  and  effectively 
communicating  your  program  scope  issues,  and  status,  and  understanding  each  stakeholder's 
perspective 

Integrated  well  led  team  with  access  to  engineering  and  systems  analysis  expertise 
Taking  advantage  of  but  not  being  driven  by  technology 
Adequate  validation  of  incrementally  delivered  products 
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Appendix  J,  CSF  Comparison  Between  GOTS,  COTS,  and  Contractor-Developed  Projects 


CSFs 

GOTS 

Rank 

COTS  Contractor 

%  of  Respondents 

GOTS  COTS  Contractor 

Raw  Data,  #  of  Respondents 
Selecting  this  CSF 

GOTS  COTS  Contractor 

Adequate  funds/resources 

1.5 

1.5 

1.5 

95.1 

100.0 

100.0 

39 

7 

9 

Clear  goals/objectives 

1.5 

7.5 

4 

95.1 

71.4 

77.8 

39 

5 

7 

Realistic  schedule 

3 

1.5 

1.5 

85.4 

100.0 

100.0 

35 

7 

9 

End  user  commitment 

4 

4.5 

3 

78.0 

85.7 

88.9 

32 

6 

8 

Clear  communication  channels 

5 

4.5 

5.5 

70.7 

85.7 

66.7 

29 

6 

6 

Having  access  to  innovative/talented 

people 

6 

9.5 

8 

68.3 

57.1 

55.6 

28 

4 

5 

Effective  management  of  risk 

7.5 

7.5 

13.5 

56.1 

71.4 

33.3 

23 

5 

3 

Support  from  senior  management 

7.5 

4.5 

10.5 

56.1 

85.7 

44.4 

23 

6 

4 

Support  from  stakeholder(s)/champion(s) 

9 

9.5 

10.5 

51.2 

57.1 

44.4 

21 

4 

4 

Effective  leadership/conflict  resolution 

10 

4.5 

8 

43.9 

85.7 

55.6 

18 

6 

5 

Having  a  clear  project  boundary 

11 

22 

8 

39.0 

0.0 

55.6 

16 

0 

5 

Effective  team  building/motivation 

12.5 

12.5 

17.5 

36.6 

28.6 

22.2 

15 

2 

2 

Recognizing  complexity 

12.5 

16.5 

13.5 

36.6 

14.3 

33.3 

15 

1 

3 

Flexible  approach  to  change 

14 

12.5 

13.5 

34.1 

28.6 

33.3 

14 

2 

3 

Effective  monitoring  and  feedback 

15 

16.5 

13.5 

31.7 

14.3 

33.3 

13 

1 

3 

Having  relevant  past  experience 

16 

16.5 

5.5 

29.3 

14.3 

66.7 

12 

1 

6 

Training  provision 

17 

11 

22 

24.4 

42.8 

0.0 

10 

3 

0 

Taking  account  of  past  experience 

18 

16.5 

17.5 

17.0 

14.3 

22.2 

7 

1 

2 

Provision  of  planning  and  control  systems 

19.5 

16.5 

22 

12.2 

14.3 

0.0 

5 

1 

0 

Taking  account  of  external  influences 

19.5 

16.5 

17.5 

12.2 

14.3 

22.2 

5 

1 

2 

Contextual  awareness 

21 

16.5 

22 

9.7 

14.3 

0.0 

4 

1 

0 

Considering  multiple  views  of  project 

22 

22 

20 

7.3 

0.0 

11.1 

3 

0 

1 

Appreciating  the  effect  of  human  error 

23 

22 

17.5 

4.9 

0.0 

22.2 

2 

0 

2 

Note.  Highlighted  numbers  are  the  top  10  CSFs  for  each  column.  Bold-faced  numbers  have  conspicuous  dil 

Terences 

.  CSFs  are  ordered 

for  GOTS. 


Appendix  K,  CSF  Comparison  Between  Project  Managers  and  Program  Managers 


CSFs 

Rank 

Project  Program 

%  of  Respondents 
Project  Program 

Raw  Data,  #  of 
Respondents 
Selecting  this  CSF 
Project  Program 

Adequate  funds/resources 

1 

2 

91.3 

92.3 

21 

24 

Clear  goals/objectives 

2.5 

3 

82.6 

88.5 

19 

23 

End  user  commitment 

2.5 

5.5 

82.6 

69.2 

19 

18 

Realistic  schedule 

4 

1 

78.3 

96.0 

18 

25 

Clear  communication  channels 

5 

5.5 

73.9 

69.2 

17 

18 

Effective  management  of  risk 

6 

8 

56.5 

61.5 

13 

16 

Effective  leadership/conflict  resolution 

8 

11.5 

52.2 

42.3 

12 

11 

Having  access  to  innovative/talented 
people 

8 

5.5 

52.2 

69.2 

12 

18 

Support  from  stakeholder(s)/champion(s) 

8 

9 

52.2 

53.8 

12 

14 

Recognizing  complexity 

10.5 

14 

47.8 

26.9 

11 

7 

Support  from  senior  management 

10.5 

5.5 

47.8 

69.2 

11 

18 

Effective  monitoring  and  feedback 

13 

18.5 

39.1 

15.4 

9 

4 

Having  relevant  past  experience 

13 

20.5 

39.1 

7.7 

9 

2 

Training  provision 

13 

16 

39.1 

23.1 

9 

6 

Having  a  clear  project  boundary 

15 

11.5 

34.8 

42.3 

8 

11 

Effective  team  building/motivation 

16.5 

11.5 

30.4 

42.3 

7 

11 

Flexible  approach  to  change 

16.5 

11.5 

30.4 

42.3 

7 

11 

Taking  account  of  past  experience 

18 

16 

21.7 

23.1 

5 

6 

Considering  multiple  views  of  project 

19 

20.5 

17.4 

7.7 

4 

2 

Appreciating  the  effect  of  human  error 

20.5 

22.5 

8.7 

3.8 

2 

1 

Taking  account  of  external  influences 

20.5 

16 

8.7 

23.1 

2 

6 

Contextual  awareness 

22.5 

22.5 

4.3 

3.8 

1 

1 

Provision  of  planning  and  control  systems 

22.5 

18.5 

4.3 

15.4 

1 

4 

Note.  Highlighted  numbers  are  the  top  10  CSFs  for  each  column.  Bold-faced  numbers  have  conspicuous 
differences.  CSFs  are  ordered  for  project  managers. 


Appendix  L,  CSF  Comparison  Between  Project  Managers  and  Program  Managers 


CSFs 

Military 

Rank 

Civil  Service 

%  of  Respondents 

Military  Civil  Service 

Raw  Data,  #  of 
Respondents 
Selecting  this  CSF 
Milit 

ary  Civil  Service 

Adequate  funds/resources 

i 

1 

100.0 

91.8 

14 

45 

Clear  goals/objectives 

2.5 

2 

92.8 

95.6 

13 

43 

Realistic  schedule 

2.5 

3 

92.8 

91.1 

13 

41 

End  user  commitment 

4 

4 

78.6 

86.7 

11 

39 

Clear  communication  channels 

Having  access  to  innovative/talented 

5 

5 

71.4 

80.0 

10 

36 

people 

6.5 

6 

57.1 

71.1 

8 

32 

Support  from  senior  management 

6.5 

9 

57.1 

57.8 

8 

26 

Effective  management  of  risk 

8.5 

9 

50.0 

57.8 

7 

26 

Recognizing  complexity 

8.5 

16 

50.0 

31.1 

7 

14 

Effective  leadership/conflict  resolution 

11.5 

9 

42.9 

57.8 

6 

26 

Flexible  approach  to  change 

11.5 

14 

42.9 

35.6 

6 

16 

Support  from  stakeholder(s)/champion(s) 

11.5 

7 

42.9 

60.0 

6 

27 

Training  provision 

11.5 

17 

42.9 

22.2 

6 

10 

Taking  account  of  past  experience 

14 

18 

35.7 

15.6 

5 

7 

Effective  monitoring  and  feedback 

15.5 

15 

28.6 

33.3 

4 

15 

Taking  account  of  external  influences 

15.5 

20.5 

28.6 

11.1 

4 

5 

Having  a  clear  project  boundary 

17.5 

11 

21.4 

48.9 

3 

22 

Having  relevant  past  experience 

17.5 

13 

21.4 

42.2 

3 

19 

Contextual  awareness 

19.5 

22.5 

14.3 

6.7 

2 

3 

Effective  team  building/motivation 

19.5 

12 

14.3 

44.4 

2 

20 

Appreciating  the  effect  of  human  error 

21.5 

22.5 

7.1 

6.7 

1 

3 

Considering  multiple  views  of  project 

21.5 

20.5 

7.1 

11.1 

1 

5 

Provision  of  planning  and  control  systems 

23 

19 

0.0 

13.3 

0 

6 

Note.  Highlighted  numbers  are  the  top  10  CSFs  for  each  column.  Bold-faced  numbers  have  conspicuous 
differences.  CSFs  are  ordered  for  military  members. 


Appendix  M ,  CSF  Comparison  Between  Software  Sustainment  and  Development 


CSFs 

Rank 

New 

Sustain  Start 

%  of  Respondents 
New 

Sustain  Start 

Raw  Data,  #  of 
Respondents 
Selecting  this  CSF 
New 

Sustain  Start 

Clear  goals/objectives 

1 

4 

94.3 

76.5 

33 

13 

Adequate  funds/resources 

2 

2 

91.4 

94.1 

32 

16 

Realistic  schedule 

3 

2 

88.6 

94.1 

31 

16 

Clear  communication  channels 

4 

7.5 

80.0 

58.8 

28 

10 

End  user  commitment 

5.5 

2 

71.4 

94.1 

25 

16 

Having  access  to  innovative/talented 
people 

5.5 

7.5 

71.4 

58.8 

25 

10 

Effective  management  of  risk 

7 

10.5 

62.9 

41.2 

22 

7 

Support  from  senior  management 

8 

5.5 

57.1 

64.7 

20 

11 

Having  a  clear  project  boundary 

9.5 

19.5 

48.6 

17.6 

17 

3 

Support  from  stakeholder(s)/champion(s) 

9.5 

9 

48.6 

47.1 

17 

8 

Effective  leadership/conflict  resolution 

11 

5.5 

45.7 

64.7 

16 

11 

Effective  team  building/motivation 

12 

17.5 

42.9 

23.5 

15 

4 

Having  relevant  past  experience 

13 

17.5 

40.0 

23.5 

14 

4 

Recognizing  complexity 

14 

14 

34.3 

29.4 

12 

5 

Effective  monitoring  and  feedback 

15 

14 

31.4 

29.4 

11 

5 

Flexible  approach  to  change 

16 

10.5 

28.6 

41.2 

10 

7 

Training  provision 

17 

14 

20.0 

29.4 

7 

5 

Taking  account  of  past  experience 

18 

14 

11.4 

29.4 

4 

5 

Considering  multiple  views  of  project 

20 

21 

8.6 

11.8 

3 

2 

Contextual  awareness 

20 

22.5 

8.6 

5.9 

3 

1 

Provision  of  planning  and  control  systems 

20 

22.5 

8.6 

5.9 

3 

1 

Taking  account  of  external  influences 

22 

14 

2.9 

29.4 

1 

5 

Appreciating  the  effect  of  human  error 

23 

19.5 

0.0 

17.6 

0 

3 

Note.  Highlighted  numbers  are  the  top  10  CSFs  for  each  column.  Bold-faced  numbers  have  conspicuous 
differences.  CSFs  are  ordered  for  sustaining  a  legacy  system. 


Appendix  N,  CSF  Comparison  Between  Project  Managers  with  5  to  10  Years  and  15+  Years  of  Experience 


CSFs 

Rank 
5  <  yrs  < 

10 

15+ 

yrs 

%  of  Respondents 

5  <  yrs  <  15+ 

10  yrs 

Raw  Data,  #  of 
Respondents 
Selecting  this  CSF 
5<yrs<  15+ 

10  yrs 

Clear  goals/objectives 

1.5 

2.5 

100.0 

84.2 

13 

16 

End  user  commitment 

1.5 

4.5 

100.0 

73.7 

13 

14 

Realistic  schedule 

3 

2.5 

92.3 

84.2 

12 

16 

Adequate  funds/resources 

Having  access  to  innovative/talented 

4 

1 

84.6 

100.0 

11 

19 

people 

5 

4.5 

76.9 

73.7 

10 

14 

Clear  communication  channels 

7 

6 

61.5 

68.4 

8 

13 

Effective  leadership/conflict  resolution 

7 

7.5 

61.5 

57.9 

8 

11 

Support  from  stakeholder(s)/champion(s) 

7 

9 

61.5 

52.6 

8 

10 

Support  from  senior  management 

9 

7.5 

53.8 

57.9 

7 

11 

Effective  management  of  risk 

10 

14.5 

46.2 

36.8 

6 

7 

Having  relevant  past  experience 

11 

11.5 

38.5 

42.1 

5 

8 

Having  a  clear  project  boundary 

12.5 

11.5 

30.8 

42.1 

4 

8 

Recognizing  complexity 

12.5 

14.5 

30.8 

36.8 

4 

7 

Effective  monitoring  and  feedback 

15 

11.5 

23.1 

42.1 

3 

8 

Effective  team  building/motivation 

15 

11.5 

23.1 

42.1 

3 

8 

Flexible  approach  to  change 

15 

16.5 

23.1 

31.6 

3 

6 

Appreciating  the  effect  of  human  error 

18.5 

21.5 

15.4 

5.3 

2 

1 

Contextual  awareness 

18.5 

21.5 

15.4 

5.3 

2 

1 

Provision  of  planning  and  control  systems 

18.5 

18.5 

15.4 

10.5 

2 

2 

Taking  account  of  past  experience 

18.5 

18.5 

15.4 

10.5 

2 

2 

Considering  multiple  views  of  project 

22 

21.5 

7.7 

5.3 

1 

1 

Taking  account  of  external  influences 

22 

21.5 

7.7 

5.3 

1 

1 

Training  provision 

22 

16.5 

7.7 

31.6 

1 

6 

Note.  Highlighted  numbers  are  the  top  10  CSFs  for  each  column.  Bold-faced  numbers  have  conspicuous 
differences.  CSFs  are  ordered  for  project  managers  with  5  up  to  10  years  of  experience. 


Appendix  O,  CSF  Comparison  Between  Successful  Projects  and  Critical  Projects 


CSFs 

Rank 

Success  Critical 

%  of  Respondents 
Success  Critical 

Raw  Data,  #  of 
Respondents 
Selecting  this  CSF 
Success  Critical 

Adequate  funds/resources 

1 

3 

100.0 

87.5 

22 

21 

Clear  goals/objectives 

2 

3 

86.4 

87.5 

19 

21 

Realistic  schedule 

3 

1 

81.8 

91.7 

18 

22 

Clear  communication  channels 

Having  access  to  innovative/talented 

4 

5 

77.3 

75.0 

17 

18 

people 

5 

6 

72.7 

70.8 

16 

17 

End  user  commitment 

6 

3 

68.2 

87.5 

15 

21 

Effective  leadership/conflict  resolution 

7 

9 

59.1 

58.3 

13 

14 

Effective  management  of  risk 

8.5 

10 

54.5 

45.8 

12 

11 

Effective  team  building/motivation 

8.5 

15 

54.5 

25.0 

12 

6 

Flexible  approach  to  change 

11 

13.5 

40.9 

29.2 

9 

7 

Having  a  clear  project  boundary 

11 

11 

40.9 

37.5 

9 

9 

Support  from  stakeholder(s)/champion(s) 

11 

7.5 

40.9 

66.7 

9 

16 

Effective  monitoring  and  feedback 

13.5 

16.5 

36.4 

20.8 

8 

5 

Support  from  senior  management 

13.5 

7.5 

36.4 

66.7 

8 

16 

Having  relevant  past  experience 

15.5 

12 

31.8 

33.3 

7 

8 

Recognizing  complexity 

15.5 

16.5 

31.8 

20.8 

7 

5 

Training  provision 

17 

13.5 

27.3 

29.2 

6 

7 

Taking  account  of  past  experience 

18 

19 

18.2 

12.5 

4 

3 

Contextual  awareness 

19 

22 

13.6 

8.3 

3 

2 

Considering  multiple  views  of  project 

20.5 

22 

9.1 

8.3 

2 

2 

Taking  account  of  external  influences 

20.5 

19 

9.1 

12.5 

2 

3 

Provision  of  planning  and  control  systems 

22 

19 

4.5 

12.5 

1 

3 

Appreciating  the  effect  of  human  error 

23 

22 

0.0 

8.3 

0 

2 

Note.  Highlighted  numbers  are  the  top  10  CSFs  for  each  column.  Bold-faced  numbers  have  conspicuous 
differences.  CSFs  are  ordered  for  project  managers  with  successful  projects. 
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